job queue implementation for python
Solution 1
In addition to multiprocessing there's also the Celery project, if you're using Django.
Solution 2
Pyres is a resque clone built in python. Resque is used by Github as their message queue. Both use Redis as the queue backend and provide a web-based monitoring application.
http://binarydud.github.com/pyres/intro.html
Solution 3
There's also "bucker" by Sylvain Hellegouarch which you can find here:
It describes itself like this:
- bucker is a queue system that supports multiple storage for the queue (memcached, Amazon SQS for now) and is driven by XML messages sent over a TCP connections between a client and the queue server.
Solution 4
If you think that Celery is too heavy for your needs then you might want to look at the simple distributed task queue:
Solution 5
redqueue? It's implemented in python+tornado framework, speaks memcached protocol and is optionally persistent into log files. Currently it is also able to behave like beanstalkd, the reserve/delete way in memcache protocol as well.
Related videos on Youtube
Admin
Updated on April 20, 2022Comments
-
Admin about 2 years
Do you know/use any distributed job queue for python? Can you share links or tools
-
Admin over 14 yearsThanks for the link? Is it strictly used with django? can we use it for standard python projects?
-
Vinay Sajip over 14 yearsI don't see why not, with suitable adaptation (not sure how much work that'll be - depends on your exact requirement).
-
Ed_WFerreira over 14 yearsCelery has an underlying library, called Carrot, that you can use without Django.
-
Admin over 14 yearsAt present, I have been using celery. I will look at the redqueue. Thanks for answer
-
asksol about 14 yearsBoth Celery and Carrot works without Django. Or that is, you can use it from outside of a Django project. Recently someone even implemented paste support: bitbucket.org/twillis/celery-paste
-
Matthew Wilcoxson over 11 yearsCelery is now designed to be used outside of Django (but still has Django support if you need it)