Pros and cons to use Celery vs. RQ

Question

Currently I'm working on python project that requires implement some background jobs (mostly for email sending and heavily database updates). I use Redis for task broker. So in this point I have two candidates: Celery and RQ. I had some experience with these job queues, but I want to ask you guys to share you experience of using this tools. So.

1. What pros and cons to use Celery vs. RQ.
2. Any examples of projects/task suitable to use Celery vs. RQ.

Celery looks pretty complicated but it's full featured solution. Actually I don't think that I need all these features. From other side RQ is very simple (e.g configuration, integration), but it seems that it lacks some useful features (e.g task revoking, code auto-reloading)

Solution

Here is what I have found while trying to answer this exact same question. It's probably not comprehensive, and may even be inaccurate on some points.

In short, RQ is designed to be simpler all around. Celery is designed to be more robust. They are both excellent.

• Documentation. RQ's documentation is comprehensive without being complex, and mirrors the project's overall simplicity - you never feel lost or confused. Celery's documentation is also comprehensive, but expect to be re-visiting it quite a lot when you're first setting things up as there are too many options to internalize

• Monitoring. Celery's Flower and the RQ dashboard are both very simple to setup and give you at least 90% of all information you would ever want

• Broker support. Celery is the clear winner, RQ only supports Redis. This means less documentation on "what is a broker", but also means you cannot switch brokers in the future if Redis no longer works for you. For example, Instagram considered both Redis and RabbitMQ with Celery. This is important because different brokers have different guarantees e.g. Redis cannot (as of writing) guarantee 100% that your messages are delivered.

• Priority queues. RQs priority queue model is simple and effective - workers read from queues in order. Celery requires spinning up multiple workers to consume from different queues. Both approaches work

• OS Support. Celery is the clear winner here, as RQ only runs on systems that support fork e.g. Unix systems

• Language support. RQ only supports Python, whereas Celery lets you send tasks from one language to a different language

• API. Celery is extremely flexible (multiple result backends, nice config format, workflow canvas support) but naturally this power can be confusing. By contrast, the RQ api is simple.

• Subtask support. Celery supports subtasks (e.g. creating new tasks from within existing tasks). I don't know if RQ does

• Community and Stability. Celery is probably more established, but they are both active projects. As of writing, Celery has ~3500 stars on Github while RQ has ~2000 and both projects show active development

In my opinion, Celery is not as complex as its reputation might lead you to believe, but you will have to RTFM.

So, why would anyone be willing to trade the (arguably more full-featured) Celery for RQ? In my mind, it all comes down to the simplicity. By restricting itself to Redis+Unix, RQ provides simpler documentation, simpler codebase, and a simpler API. This means you (and potential contributors to your project) can focus on the code you care about, instead of having to keep details about the task queue system in your working memory. We all have a limit on how many details can be in our head at once, and by removing the need to keep task queue details in there RQ lets get back to the code you care about. That simplicity comes at the expense of features like inter-language task queues, wide OS support, 100% reliable message guarantees, and ability to switch message brokers easily.

Comments

[Jim]

I totally disagree with your assessment. I struggled for a couple of weeks to get Celery running successfully on my Debian server, even after reading much of the documentation and numerous blog posts. The main problem I had was that if you got something wrong in your configuration, Celery wouldn't provide any feedback on what the problem might be. And when I finally did get it working, I started getting some type os OSError deep down in the Celery stack. I posted a issue on Github but no one could help. I wouldn't touch Celery again with a ten-foot pole.

[MiniGunnR]

Effin' OSError man. No such file or directory. I have no clue where to start. I'll try out RQ for the first time tonight.

[eserdk]

Subtask support. Celery supports subtasks (e.g. creating new tasks from within existing tasks). I don't know if RQ does As for 24.05.2019 RQ supports subtasks (inner call for queue) too.

[Kostas Mouratidis]

One minor note: RQ supports passing your own connection class so if you want to implement the interface of python's redis library (i.e. the Redis, Pipeline, PubSub classes) then you can have whatever broker you want.