How to use asyncio: Python’s built-in async library

Python’s asynchronous programming functionality, or async for short, allows you to write programs that get more work done by not waiting for independent tasks to finish. The asyncio library is a set of tools included with Python for asynchronously processing disk or network I/O.

asyncio provides two kinds of APIs for dealing with asynchronous operations: high-level and low-level. The high-level APIs are the most generally useful, and applicable to the widest variety of applications. The low-level APIs are powerful, but also complex and less frequently used.

In this article, we’ll walk through asyncio‘s most frequently used high-level APIs, demonstrating how to use them for common operations involving asynchronous tasks.

If you’re completely new to asynchronous programming in Python, or you could use a refresher on how it works, read my introduction to Python async before diving in here.

Run coroutines and tasks in Python

Naturally, the most common use for asyncio is to run the asynchronous parts of your Python script. This means learning to work with coroutines and tasks.

Python’s async components, including coroutines and tasks, can only be used with other async components, and not with conventional synchronous Python, so you need asyncio to bridge the gap. To do this, you use the asyncio.run function:

import asyncio
async def main():
print ('Waiting 5 seconds. ')
for _ in range(5):
await asyncio.sleep(1)
print ('.')
print ('Finished waiting.')
asyncio.run(main())


This runs main(), along with any coroutines main() fires off, and waits for a result to return.

As a general rule, a Python program should have only one.run() statement, just as a Python program should have only one main() function. Async, if used carelessly, can make the control flow of a program hard to read. Having a single entry point to a program’s async code keeps things from getting hairy.

Async functions can also be scheduled as tasks, or objects that wrap coroutines and help run them.

async def my_task():
do_something()
task = asyncio.create_task(my_task())



my_task() is then run in the event loop, with its results stored in task.

Tasks are useful if you want to set something up to run in the background and then check for a result later. For instance, this task could be stored in a list and checked at some point in the future, either on a schedule or on demand.

If you have only one task you want to get results from, you can use asyncio.wait_for(task) to wait for the task to finish, then use task.result() to retrieve its result. But if you’ve scheduled a few tasks to execute and you want to wait for all of them to finish before continuing to something else, use asyncio.wait([task1, task2]) to gather the results. You can also set a timeout for the operations if you don’t want them to run past a certain length of time.

Manage an async event loop in Python

Another common use for asyncio is to manage the async event loop. The event loop is an object that runs async functions and callbacks; it’s created automatically when you use asyncio.run(). You generally want to use only one async event loop per program, again to keep things manageable.

If you’re writing more advanced software, such as a server, you’ll need lower-level access to the event loop. To that end, you can “lift the hood” and work directly with the event loop’s internals. But for simple jobs, you won’t need to.

Read and write data with streams in Python

The best scenarios for async are long-running network operations, where the application may block waiting for some other resource to return a result. To that end, asyncio offers streams, which are high-level mechanisms for performing network I/O. This includes acting as a server for network requests.

asyncio uses two classes, StreamReader and StreamWriter, to read and write from the network at a high level. If you wanted to read from the network, you would use asyncio.open_connection() to open the connection. That function returns a tuple of StreamReader and StreamWriter objects, and you would use.read() and.write() methods on each to communicate.

To receive connections from remote hosts, use asyncio.start_server(). The asyncio.start_server() function takes as an argument a callback function, client_connected_cb, which is called whenever it receives a request. That callback function takes instances of StreamReader and StreamWriter as arguments, so you can handle the read/write logic for the server. (See here for an example of a simple HTTP server that uses the asyncio-driven aiohttp library.)

Synchronize tasks in Python

Asynchronous tasks tend to run in isolation, but sometimes you will want them to communicate with each other. asyncio provides several mechanisms for synchronizing between tasks:

Queues: asyncio queues allow asynchronous functions to line up Python objects to be consumed by other async functions—for instance, to distribute workloads between different kinds of functions based on their behaviors.

Synchronization primitives: Locks, events, conditions, and semaphores in asyncio work like their conventional Python counterparts. They’re for when you want to coordinate other kinds of activity between tasks, like waiting for a larger condition to be true, or for taking control of a resource that can only be used by one entity at a time.

One thing to keep in mind is that none of these methods is thread-safe. This isn’t an issue for async tasks running in the same event loop. But if you’re trying to share information with tasks in a different event loop, operating system thread, or process, you’ll need to use the threading module and its objects.

Further, if you want to launch coroutines across thread boundaries, use the asyncio.run_coroutine_threadsafe() function, and pass the event loop to use with it as a parameter.

Pause a coroutine in Python

Another common use of asyncio, which is not often discussed, is waiting for some arbitrary length of time inside a coroutine. You can’t use time.sleep() for this, or you’ll block the entire program. Instead, use asyncio.sleep(), which allows other coroutines to continue running.

Now, if you’re thinking of using asyncio.sleep() in a loop to wait constantly for some external condition … don’t. While you can do this, in theory, it’s a clumsy way to handle that situation. Passing an asyncio.Event object to a task is a better approach, allowing you to just wait for the Event object to change.

Async and file I/O

Network I/O in async can be made not to block, as described above. But local file I/O blocks the current thread by default. One workaround is to delegate the file I/O operation to another thread using asyncio.to_thread(), so that other tasks in the event loop can still be processed.

Another way to handle file I/O in async is with the third-party aiofiles library. This gives you high-level async constructs for opening, reading, and writing files—e.g., async with aiofiles.open('myfile.txt') as f:. If you don’t mind having it as a dependency in your project, it’s an elegant way to deal with this issue.

Use lower-level async in Python

Finally, if you think that the app you’re building may require asyncio’s lower-level components, take a look around before you start coding. There’s a good chance someone has already built an async-powered Python library that does what you need.

For instance, if you need async DNS querying, check the aiodns library, and for async SSH sessions, there’s asyncSSH. Search PyPI by the keyword “async” (plus other task-related keywords), or check the hand-curated Awesome Asyncio list for ideas.