Trying to build an async task pool in python using asyncio. I am trying to limit cocurrrent tasks and be able to handle errors/exceptions.

I am using code below to run task to raise and error in task 2 (inside worker task id 2 raises error i want it to raise error). I am trying to make task 3 is executed even if task 2 raises error but this code completes task 0 and 1 but does not start task 3. I am expecting it to move to next task. How can i make it move to next tasks even if an exception is raised?

import asyncio
import asyncio.exceptions

async def worker(semaphore, task_id):
    async with semaphore:
        try:
            if task_id == 2:
                raise ValueError("Something went wrong in task 2")
            await asyncio.sleep(1)  
            print(f"Task {task_id} completed")
            return f"Result from task {task_id}"
        except ValueError as e:
            print(f"Task {task_id} caught exception: {e}")
            return None  #

async def main():
    semaphore = asyncio.Semaphore(2)  
    tasks = [worker(semaphore, i) for i in range(4)]
    results = await asyncio.gather(*tasks, return_exceptions=True) 
    for i, result in enumerate(results):
      if isinstance(result, Exception):
        print(f"Task {i} raised an exception {result}")
      else:
        print(f"Task {i} returned {result}")

if __name__ == "__main__":
    asyncio.run(main())

Trying to build an async task pool in python using asyncio. I am trying to limit cocurrrent tasks and be able to handle errors/exceptions.

I am using code below to run task to raise and error in task 2 (inside worker task id 2 raises error i want it to raise error). I am trying to make task 3 is executed even if task 2 raises error but this code completes task 0 and 1 but does not start task 3. I am expecting it to move to next task. How can i make it move to next tasks even if an exception is raised?

import asyncio
import asyncio.exceptions

async def worker(semaphore, task_id):
    async with semaphore:
        try:
            if task_id == 2:
                raise ValueError("Something went wrong in task 2")
            await asyncio.sleep(1)  
            print(f"Task {task_id} completed")
            return f"Result from task {task_id}"
        except ValueError as e:
            print(f"Task {task_id} caught exception: {e}")
            return None  #

async def main():
    semaphore = asyncio.Semaphore(2)  
    tasks = [worker(semaphore, i) for i in range(4)]
    results = await asyncio.gather(*tasks, return_exceptions=True) 
    for i, result in enumerate(results):
      if isinstance(result, Exception):
        print(f"Task {i} raised an exception {result}")
      else:
        print(f"Task {i} returned {result}")

if __name__ == "__main__":
    asyncio.run(main())

• python
• python-asyncio

• The posted program works for me as I would expect. When a program behaves not like you think it should, please always post the output you are getting and what you've expected to get instead. – VPfB Commented 1 hour ago

1 Answer 1

after I remove try-catch block to make sure Task 2 raise exception, the script runs ok for me and here is output

Task 0 completed
Task 1 completed
Task 3 completed
Task 0 returned Result from task 0
Task 1 returned Result from task 1
Task 2 raised an exception Something went wrong in task 2
Task 3 returned Result from task 3

And here is the script after I modified

import asyncio
import asyncio.exceptions


async def worker(semaphore, task_id):
    async with semaphore:
        if task_id == 2:
            raise ValueError("Something went wrong in task 2")
        await asyncio.sleep(1)
        print(f"Task {task_id} completed")
        return f"Result from task {task_id}"


async def main():
    semaphore = asyncio.Semaphore(2)
    tasks = [worker(semaphore, i) for i in range(4)]
    results = await asyncio.gather(*tasks, return_exceptions=True)
    for i, result in enumerate(results):
        if isinstance(result, BaseException):
            print(f"Task {i} raised an exception {result}")
        else:
            print(f"Task {i} returned {result}")


if __name__ == "__main__":
    asyncio.run(main())