Real-life simple examples of asyncio programming in Python.
A fun and useful project for a hackerspace is to be able to share whether the space is openned or closed to the public at any given time.
At hackens, it's an idea that fermented for a while and I somehow ended up writing most of the communication code. It's 100% a plumbing project, so I'll at least try to make it worthwile.
While looking online to learn about how to use asyncio, I mostly found
examples so minimal I struggled to see how they would fit in a real project.
Wading through the documentation, with a bit of trial and error, I learned a
lot about asynchronous programming, from a client and server perspective, and I
hope to give some insight into how some elements of the asyncio module in
Python can be used in real-world examples.
I will assume you have some notion of asynchronous computation, including what is a coroutine and how it differs from regular callables, and what is an event loop. If needed there's a quick refresher at the end.
First, let me describe the overall architecture of the system. It's not too complicated, but there are just quite a few elements here. If you want to look into it further, all the code for the project is publicly available on a git forge.
The data flows downstream, from the sensors, to a central orchestrator, to local network clients and to the exterior through a proxy server.
The sensors do not do any computation, instead they send the raw data to the orchestrator through a serial connection, which parses it, interprets it, and returns a status (open/closed, forced/not forced) to the sensor box to indicate the status through a LED.
The orchestrator also publishes the status to all the registred clients. Currently, this includes an ESP-32 at the entrance of the space which turns on a LED strip when the space is open, and a proxy server. Since the ESP-32 is on the same local network, we chose to connect through a socket, and the connexion to the proxy server is done through a (authenticated) websocket.
Since there are only a few clients, the IO there is done synchronously. But, because we want to react to a change in the state without stalling the system and by sharing the change as soon as possible, I opted for an async protocol for the serial connection.
Now, the proxy server exposes one public websocket, which can have many clients connected to it. This socket is purely one-directional: clients can't make requests to the server, they have to listen. Because of this, we opted for an async server design here.
The relevant part of the project is available here.
We'll start with the proxy server. It's as simple as possible, one websocket listens to the orchestrator to get the current status, and another publishes the state.
Python's websocket module,
websockets has a very good
async workflow, taking a coroutine (that deals with one connection) and
creating another one (that deals with all connections throughout the life of
the server). It is then a matter of defining the desired behavior and
scheduling the execution of the server. In its most simple form, it is built
like this:
1 2 3 4 5 6 7 8 9 from websockets.asyncio.server import serve server = await serve( server_fn, hostname, port, **kwargs ) # Now, schedule `server.serve_forever()` for execution
where server_fn is a coroutine with the signature server_fn(connection) -> None.
Here is the important bit: nothing will not run right now. We simply defined an object containing all the info needed. In order to actually run the server, we need to:
server.serve_forever())By defining several coroutines and scheduling them as tasks (typically using
the asyncio.TaskGroup mechanism), we can have multiple servers running
concurrently.
Simply put, we have code that looks like this:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 import asyncio from websockets.asyncio.server import serve async def main(): listener = await serve( listen_fn, hostname_l, port_l, **kwargs_l ) speaker = await serve( speak_fn, hostname_s, port_s, **kwargs_s ) async with asyncio.TaskGroup() as tg: tg.create_task(listener.serve_forever()) tg.create_task(speaker.serve_forever()) asyncio.run(main())
asyncio.Protocol-based subscribe/publish serverThe relevant part of the project is available here.
The orchestrator has the main challenge of dealing with serial connections, for
which we don't have as nice an interface as for websockets. The module we use
is serial_asyncio. If you
don't already know what you're doing, the documentation can be quite
lackluster, basically telling you to write your own
protocol (factory).
While it's presented mainly as a way to handle TCP/UDP/whatever network
protocol you want, at its core, an asyncio.Protocol is a way to do callbacks
on a connection. The actual detail of how the data is sent is not the job of
the protocol, but of its associated asyncio.Transport object, which will not
be covered here. The only thing you need to know about the Transport objects
is that they have a reference to the underlying event loop and handle all the
scheduling of I/O tasks, including of the callbacks defined in the Protocol.
Since all the
loop.call_{soon,later,...}methods take a callable (instead of the coroutines we're used to now), the callbacks defined inProtocolare indeed functions, despite still being part of an asynchronous I/O pipeline. {: .prompt-info }
By writing your own protocol, you can customize the behavior when:
connection_made)data_received, eof_received)pause_writing, resume_writing)connection_lost)For us, the purpose of the protocol we implement is to:
Basically, we want to override the data_received method and follow the
recipe given by serial_asyncio. So, that's what we do:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 import asyncio import serial_asyncio ############################################################################### # Defining the protocol class MyProtocol(asyncio.Protocol): """Splits data per line and processes each line individually.""" def __init__(self): self.buffer = "" def data_recieved(self, data: bytes): self.buffer += str(data, encoding="utf-8") if "\n" in self.chunk: messages = self.chunk.split("\n") self.chunk = messages[-1] self.process(messages[:-1]) def process(self, messages: list[str]): # Do stuff here ############################################################################### # Instanciating and running the protocol # Replace with asyncio.get_event_loop if needed event_loop = asyncio.new_event_loop() async def reader(): transport, protocol = await serial_asyncio.create_serial_connection( event_loop, MyProtocol, serial_port, baudrate=115200, ) event_loop.run_until_complete(reader()) event_loop.close()
Now, we have the structure of the reader server. But how do we trigger
side-effects? Well, serial_asyncio.create_serial_connection takes a protocol
factory. So, we can just hide any and all arguments inside a lambda, with
something that looks like:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 class MyProtocol(asyncio.Protocol): def __init__(self, arg): self.thing = arg def process(self, messages): self.thing.do_whatever(messages) ... transport, protocol = await serial_asyncio.create_serial_connection( event_loop, lambda: MyProtocol(arg), serial_port, baudrate=115200, )
In practice, we have a State class that stores the current state and signals
the clients to broadcast whenever a change occurs. The specifics of broadcaster
depends on which clients are registered and don't really matter here.
If you're curious, the concrete implementation is
here
So, in the end, we have:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 import asyncio import serial_asyncio ############################################################################### # Defining the state keeper class State: def __init__(self, broadcaster: Broadcaster): self._current_status = OutsideStatus.CLOSED self.broadcaster = broadcaster @property def current_status(self) -> OutsideStatus: return self._current_status @current_status.setter def current_status(self, val) -> None: if self._current_status != val: self._current_status = val self.broadcaster.broadcast(self._current_status) ############################################################################### # Defining the protocol class MyProtocol(asyncio.Protocol): """Splits data per line and processes each line individually.""" def __init__(self, state: State): self.buffer = "" self.state = state def data_recieved(self, data: bytes): self.buffer += str(data, encoding="utf-8") if "\n" in self.chunk: messages = self.chunk.split("\n") self.chunk = messages[-1] self.process(messages[:-1]) def process(self, messages: list[str]): new_status = self.get_stats_from_messages(messages) self.state.current_status = new_status ############################################################################### # Instanciating and running the protocol # Replace with asyncio.get_event_loop if needed event_loop = asyncio.new_event_loop() state = State(broadcaster = ...) async def reader(): transport, protocol = await serial_asyncio.create_serial_connection( event_loop, lambda: MyProtocol(state), serial_port, baudrate=115200, ) event_loop.run_until_complete(reader()) event_loop.close()
Now, I mentionned that, because in our case we know the subscribed clients, that there are only a handful of them, we do not need to have the broadcast itself be asynchronous.
If we wanted to go a step further and actually have the broadcast itself be asynchronous, I think a way to do it would be to:
self.broadcaster.broadcast be a coroutine (details depend on the
underlying connection),event_loop object to the stateself.broadcaster.broadcast(...) by
event_loop.create_task(self.broadcaster.broadcast(...)This was a very technical article, more than I usually like to write, but I
figured some simple but actually useful examples of using asyncio in
different contexts are always useful to have around.
Writing the code for this project, I had to rethink the architecture quite a few times to accomodate for the async pipeline, which is not something I usually care for. Hopefully, with these examples in mind, you can get some idea of how to structure a project around these IO tasks, whether by having a server-based or a protocol-based architecture.
Traditionally, we think of programs as a list of instructions being ran from top to bottom, one after another, in a well specified order. This has the benefit of being easier to write and more predictible. But it has the drawback of potentially waiting on IO bound tasks.
Asynchronous programming, for the most part, is designed around finding a way not to wait while there is still something else to be done. A typical example is a web server. If a server has several clients to serve, you could:
Or, you could reorder the operations and:
In other words, if you have two letters to send, you can either wait for the delivery notice before sending the next one (synchronous case) or send both letters at the same time and wait only once for the delivery notices. In particular, if something went wrong and the first letter took forever to deliver, we don't have to wait to send the second letter, and both letter delivery times are uncorrelated.
But how do we do this in practice? Well, we schedule, much like an operating system schedules which process to run at any given CPU cycle. Essentially, our program is a big loop that looks like this pseudocode:
1 2 3 while True: new_task = select_task_to_run() execute_a_bit(new_task)
That loop is called an event loop. It's the brain of the operations. You, as the programmer, will interact with this loop by creating coroutines and scheduling them.
A coroutine is a partially ran function, and is defined like so:
1 2 3 4 5 6 async def send_letter(): print("Sending letter.") await async.time(1) print("Letter has been received!") coro = send_letter() # <-- this is a coroutine
When defined, a coroutine doesn't do anything. It's because it has not been scheduled yet. To schedule a coroutine, you can do it in several ways:
async block using the keyword awaitasyncio.run or low-level
methods of the event loop.The await keyword is a fence, the execution can't go further without having
executed what's behind. You ask the program to wait until the instruction is
finished to continue. But this doesn't mean that unrelated tasks can't run at
the same time, and that's the magic of it!
If we continue our example, send_letter prints a first message and waits for
1 second before printing a message. If we only run this once, it's not much
different from having a synchronous code. However, if we run it several times
in quick succession, it's like the letter example, we will send multiple
executions at once and all the prints will arrive more or less at the same
time.
1 2 3 4 5 6 7 8 async def main(): coros = [send_letter() for _ in range(5)] tasks = [asyncio.create_task(coro) for coro in coros] for t in tasks: await t if __name__ == "__main__": asyncio.run(main())
If we run this code, the output will be:
1 2 3 4 5 6 7 8 9 10 Sending letter. Sending letter. Sending letter. Sending letter. Sending letter. Letter has been recieved! Letter has been recieved! Letter has been recieved! Letter has been recieved! Letter has been recieved!
What we did here is, first, create a new coroutine (main) which:
main (asyncio.run).Under the hood, asyncio manages the event loop so you don't have to think
about it and have a nicer interface to work with.
That's all for the refresher on async programming in Python, now go back to the article!
by CelestialSynapse