Celery: per task concurrency limits (# of workers per task)?

Question

Is it possible to set the concurrency (the number of simultaneous workers) on a per-task level in Celery? I'm looking for something more fine-grained that CELERYD_CONCURRENCY (that sets the concurrency for the whole daemon).

The usage scenario is: I have a single celerlyd running different types of tasks with very different performance characteristics - some are fast, some very slow. For some I'd like to do as many as I can as quickly as I can, for others I'd like to ensure only one instance is running at any time (ie. concurrency of 1).

Solution

You can use automatic routing to route tasks to different queues which will be processed by celery workers with different concurrency levels.

celeryd-multi start fast slow -c:slow 3 -c:fast 5

This command launches 2 celery workers listening fast and slow queues with 3 and 5 concurrency levels respectively.

CELERY_ROUTES = {"tasks.a": {"queue": "slow"}, "tasks.b": {"queue": "fast"}}

The tasks with type tasks.a will be processed by slow queue and tasks.b tasks by fast queue respectively.

Comments

[nosid]

Thanks 0x00mh. So I can define concurrency on queues, but not on tasks. And I believe this means I'm starting multiple celery daemons. So I guess that means there's no way to set the per-task concurrency without using a separate daemon?

[mher]

CELERYD_CONCURRENCY defines how many processes (worker processes) to launch. Worker processes consume messages from the broker independently. Message contains the task name to execute.