Gracefully Restarting Django Celery Instances

Question

I'm wondering what the best way to restart Celery is without losing any tasks during the restart.

I'm currently running it as a daemon using the init.d/ script provided with celery - everything is working great.

Nonetheless, I will need to restart it (I think?) in order for it to see code updates. My concern is if someone uploads a photo and we want to do processing on it during that 5s or so window that celery is restarting the task will be lost forever and we'll start seeing weird issues.

Is there a suggested way to reload code for celery without losing any transactions in the meantime?

Thanks,

Phil

Solution

You can safely restart celery worker without losing any tasks with the TERM signal. So the celery will end the current tasks and die.

If you want your tasks to be retried if something goes wrong you can also use the acks_late option (Task.acks_late / CELERY_ACKS_LATE).

Q: How can I safely shut down the worker? (from: http://celery.readthedocs.org/en/latest/faq.html#how-can-i-safely-shut-down-the-worker)

Answer: Use the TERM signal, and the worker will finish all currently executing jobs and shut down as soon as possible. No tasks should be lost.

You should never stop worker with the KILL signal (-9), unless you’ve tried TERM a few times and waited a few minutes to let it get a chance to shut down. As if you do tasks may be terminated mid-execution, and they will not be re-run unless you have the acks_late option set (Task.acks_late / CELERY_ACKS_LATE).

To stop the worker (from: http://celery.readthedocs.org/en/latest/userguide/workers.html#worker-stopping)

Q: Stopping the worker

Shutdown should be accomplished using the TERM signal.

To send a TERM signal, use the linux command KILL (from: http://linux.about.com/od/commands/l/blcmdl1_kill.htm)

kill - terminate a process SYNOPSIS

kill [ -s signal | -p ] [ -a ] [ -- ] pid ... kill -l [ signal ] DESCRIPTION

The command kill sends the specified signal to the specified process or process group. If no signal is specified, the TERM signal is sent. The TERM signal will kill processes which do not catch this signal. For other processes, it may be necessary to use the KILL (9) signal, since this signal cannot be caught.

Comments

[Admin]

If you mean the worker, see docs.celeryproject.org/en/latest/userguide/… - it'll be fine