Eventlet or gevent or Stackless + Twisted, Pylons, Django and SQL Alchemy

Question

We're using Twisted extensively for apps requiring a great deal of asynchronous io. There are some cases where stuff is cpu bound instead and for that we spawn a pool of processes to do the work and have a system for managing these across multiple servers as well - all done in Twisted. Works great. The problem is that it's hard to bring new team members up to speed. Writing asynchronous code in Twisted requires a near vertical learning curve. It's as if humans just don't think that way naturally.

We're considering a mixed approach perhaps. Maybe keep the xmlrpc server part and process management in Twisted and implement the other stuff in code that at least looks synchronous to some extent while not being as such. Then again I like explicit over implicit so I have to think about this a bit more. Anyway onto greenlets - how well does that stuff work? So there's Stackless and as you can see from my Gallentean avatar I'm well aware of the tremendous success in it's use for CCP's flagship EVE Online game first hand. What about Eventlet or gevent? Well for now only Eventlet works with Twisted. However gevent claims to be faster as it's not a pure python implementation but rather relies upon libevent instead. It also claims to have fewer idiosyncrasies and defects. gevent It's maintained by 1 guy as far as I can tell. This makes me somewhat leery but all great projects start this way so... Then there's PyPy - I haven't even finished reading about that one yet - just saw it in this thread: Drawbacks of Stackless.

So confusing - I'm wondering what the heck to do - sounds like Eventlet is probably the best bet but is it really stable enough? Anyone out there have any experience with it? Should we go with Stackless instead as it's been around and is proven technology - just like Twisted is as well - and they do work together nicely. But still I hate having to have a separate version of Python to do this. what to do....

This somewhat obnoxious blog entry hit the nail on the head for me though: Asynchronous IO for Grownups I don't get the Twisted is being like Java remark as to me Java is typically where you are in the threading mindset but whatever. Nevertheless if that monkey patch thing really works just like that then wow. Just wow!

Solution 1

You might want to check out:

• Comparing gevent to eventlet
• Reports from users who moved from twisted or eventlet to gevent

Eventlet and gevent are not really comparable to Stackless, because Stackless ships with a standard library that is not aware of tasklets. There are implementations of socket for Stackless but there isn't anything as comprehensive as gevent.monkey. CCP does not use bare bones Stackless, it has something called Stackless I/O which AFAIK is windows-only and was never open sourced (?).

Both eventlet and gevent could be made to run on Stackless rather than on greenlet. At some point we even tried to do this as a GSoC project but did not find a student.

Solution 2

Answering part of your question - if you look at http://speed.pypy.org you'll see that using twisted on top of PyPy may give you some speedups. This depends of course on your workload, but it's probably worth checking out.

Cheers,
fijal

Comments

[Khorkrak]

Thanks so much Denis - I'd already read through the comparison of gevent to eventlet but the Reports from users who moved from twisted or eventlet to gevent was enlightening.

[Nick Zalutskiy]

@Khorkrak It's been a year a half since you've asked this question. I was wondering if you could share your experience since then. What did you end up doing? How did it work out for you?

[Robert Zaremba]

The comparison is biassed. Eventlet has a lot of improvements since that time, and now has bigger support.

[Robert Zaremba]

Using PyPy gives you a huge speedup, so it's worth to consider. There is also Tornado which is simpler, much faster then twisted, works more stable on high load.