Which Python async library would be best suited for my code? Asyncore? Twisted?

Question

I have a program I'm working on that will be reading from two 'network sources' simultaneously. I wanted to try out an asynchronous approach rather than use threading. This has lead me to wonder which library to use...

I've come up with some simple example code that kind of demonstrates what my program will be doing:

import sniffer

def first():
    for station in sniffer.sniff_wifi():
        log(station.mac())

def second():
    for station in sniffer.sniff_ethernet():
        log(station.mac())

first()
second()

The two sniffer methods look somewhat like this:

def sniff_wifi(self):

    while True:
        yield mac_address

The while True loop obviously makes them blocking.

I want to use asyncore for this as it is part of the standard library. No 3rd party dependencies are a bonus. However, that doesn't mean I won't use it if you recommend I do...

Can I achieve what I'm trying to do with asyncore? If so, could you show me how to convert my example code to 'asyncore code'? Do you know of any good asyncore tutorials?

Solution 1

Twisted is better in pretty much every possible way. It's more portable, more featureful, simpler, more scalable, better maintained, better documented, and it can make a delicious omelette. Asyncore is, for all intents and purposes, obsolete.

It's hard to demonstrate all the ways in which Twisted is superior in a short answer (how could I demonstrate a http/dns/ssh/smtp/pop/imap/irc/xmpp/process-spawning/multi-threading server in a short example?), so instead I'll focus on one of the most common misconceptions that people seem to have about Twisted: that it's somehow more complex or harder to use than asyncore.

Let's start with an asyncore example. In order to avoid a biased presentation, I'll use an example from someone else who still likes asyncore a bit. Here's a simple asyncore example taken from Richard Jones' weblog (with comments elided for brevity).

First, here's the server:

import asyncore, socket

class Server(asyncore.dispatcher):
    def __init__(self, host, port):
        asyncore.dispatcher.__init__(self)
        self.create_socket(socket.AF_INET, socket.SOCK_STREAM)
        self.bind(('', port))
        self.listen(1)

    def handle_accept(self):
        socket, address = self.accept()
        print 'Connection by', address
        EchoHandler(socket)

class EchoHandler(asyncore.dispatcher_with_send):
    def handle_read(self):
        self.out_buffer = self.recv(1024)
        if not self.out_buffer:
            self.close()

s = Server('', 5007)
asyncore.loop()

and here's the client:

import asyncore, socket

class Client(asyncore.dispatcher_with_send):
    def __init__(self, host, port, message):
        asyncore.dispatcher.__init__(self)
        self.create_socket(socket.AF_INET, socket.SOCK_STREAM)
        self.connect((host, port))
        self.out_buffer = message

    def handle_close(self):
        self.close()

    def handle_read(self):
        print 'Received', self.recv(1024)
        self.close()

c = Client('', 5007, 'Hello, world')
asyncore.loop()

There are a few obscure cases that this code doesn't handle correctly, but explaining them is boring and complicated, and the code has already made this answer long enough.

Now, here's some code that does basically the same thing, with Twisted. First, the server:

from twisted.internet import reactor, protocol as p

class Echo(p.Protocol):
    def dataReceived(self, data):
        self.transport.write(data)

class EchoFactory(p.Factory):
    def buildProtocol(self, addr):
        print 'Connection by', addr
        return Echo()

reactor.listenTCP(5007, EchoFactory())
reactor.run()

And now, the client:

from twisted.internet import reactor, protocol as p

class EchoClient(p.Protocol):
    def connectionMade(self):
        self.transport.write(self.factory.data)

    def dataReceived(self, data):
        print 'Received:', data
        self.transport.loseConnection()

class EchoClientFactory(p.ClientFactory):
    protocol = EchoClient
    def __init__(self, data):
        self.data = data

reactor.connectTCP('localhost', 5007, EchoClientFactory('Hello, world'))
reactor.run()

There are a couple of things that I'd like to draw your attention to. First of all, the Twisted example is 25% shorter, even for something this trivial. 40 lines for asyncore, only 30 for Twisted. As your protocol grows more complex, this difference will get bigger and bigger, as you need to write more and more support code for asyncore that would have been provided for you by Twisted.

Second, Twisted provides a complete abstraction. With the asyncore example, you have to use the socket module to do the actual networking; asyncore provides only multiplexing. This is a problem if you need portable behavior on platforms such as Windows. It also means that asyncore completely lacks facilities for doing asynchronous sub-process communication on other platforms; you can't stuff arbitrary file descriptors into a select() call on Windows.

Third, the Twisted example is transport neutral. None of Echo and EchoFactory and EchoClient and EchoClientFactory is in any way specific to TCP. You can make these classes into a library that can be connected via SSH, or SSL, or a UNIX socket, or a pipe, only by changing the one connectTCP/listenTCP call at the bottom. This is important, as supporting something like TLS directly in your protocol logic is very tricky. For example, a 'write' in TLS will trigger a 'read' at the lower level. So, you need to separate these concerns out to get them right.

Finally, specific to your use-case, if you're dealing with MAC addresses and ethernet frames directly, Twisted contains Twisted Pair, a low-level library for dealing with IP and ethernet-level networking. This isn't the most actively maintained part of Twisted; the code is quite old. But, it should work, and if it doesn't we will take any bugs in it seriously and (eventually) see that they get fixed. As far as I'm aware, there's no comparable library for asyncore, and it certainly doesn't contain any such code itself.

Solution 2

Asyncore is nice but not very feature rich so you might run into problems later when your app grows. That being said, it's great to prototype stuff. The approach is quite simple. You define methods to handle certain events in a class (when read is possible, when write is possible etc) and then subclass that from the asyncore.dispatcher (I think) class.

The official docs for the module as well as Doug Hellmann's excellent PyMOTW article on it are good sources to check out for docs and examples.

If your protocol is conversational (e.g. send this, receive that), you can check out the asynchat module also distributed with the standard library for ideas.

Twisted is a much more heavy duty approach. I'm sure it will work better for larger projects given how much it's used but I can't say anything more because I don't have any first hand experience with it.