How do I restart celery workers gracefully?

Question

While issuing a new build to update code in workers how do I restart celery workers gracefully?

Edit: What I intend to do is to something like this.

• Worker is running, probably uploading a 100 MB file to S3
• A new build comes
• Worker code has changes
• Build script fires signal to the Worker(s)
• Starts new workers with the new code
• Worker(s) who got the signal after finishing the existing job exit.

Solution 1

According to https://docs.celeryq.dev/en/stable/userguide/workers.html#restarting-the-worker you can restart a worker by sending a HUP signal

 ps auxww | grep celeryd | grep -v "grep" | awk '{print $2}' | xargs kill -HUP

Solution 2

celery multi start 1 -A proj -l info -c4 --pidfile=/var/run/celery/%n.pid
celery multi restart 1 --pidfile=/var/run/celery/%n.pid

http://docs.celeryproject.org/en/latest/userguide/workers.html#restarting-the-worker

Solution 3

If you're going the kill route, pgrep to the rescue:

kill -9 `pgrep -f celeryd`

Mind you, this is not a long-running task and I don't care if it terminates brutally. Just reloading new code during dev. I'd go the restart service route if it was more sensitive.

Solution 4

You can do:

celery multi restart w1 -A your_project -l info  # restart workers

Example

Solution 5

You should look at Celery's autoreloading

View more solutions

Comments

[Oleksandr]

This seems to be experimental This is an experimental feature intended for use in development only, using auto-reload in production is discouraged as the behavior of reloading a module in Python is undefined

[Oleksandr]

sudo ps auxww | grep celeryd | grep -v "grep" | awk '{print $2}' | sudo xargs kill -HUP exclude grep :-)

[chanux]

You can replace grep celeryd | grep -v "grep" with grep [c]eleryd. Just saying.

[mennanov]

It seems that it is not a graceful restart, is it? As the docs say: "Other than stopping then starting the worker to restart, you can also restart the worker using the HUP signal, but note that the worker will be responsible for restarting itself so this is prone to problems and is not recommended in production" So what is the best way to reload Celery in production to avoid failures?

[Thorin Schiffer]

Be also aware, that turning on this flag can cause orphan processes running, which in its turn causes result backend working unproperly

[orzel]

(pkill does this in a cleaner way)

[webjunkie]

Uugh... it says right there "The easiest way to manage workers for development is by using celery multi. For production deployments you should be using init scripts or other process supervision systems". This answer does not apply to running in production!

[webjunkie]

For celery multi: "For production deployments you should be using init scripts or other process supervision systems". As for HUP: "this is prone to problems and is not recommended in production"

[zengr]

@webjunkie The OP didn't say "in product deployment", so not sure why would you downvote it if it was not mentioned in the original question.

[webjunkie]

He also did not say he wants a solution for an e.g. testing environment. A lot of people will not bother to read more and dangerously go and use a solution that appears right to them. So it is only fair to mention drawbacks and not simply copy and paste something from a documentation ignoring notes and stripping away further recommendations.

[Mark Chackerian]

The autoreload options seems to have been removed from Celery 4

[JL Peyret]

didn't know that. I still prefer seeing a list of processes that will be killed beforehand however: step 1 - tune your pgrep, step 2 weaponize it by feeding it to the kill.

[GDorn]

The celery documentation appears to be self-conflicting on this subject; here it says don't use celery multi in production, but in the daemonization section the example systemd config file uses celery multi.

[egor83]

Unit celery.service could not be found.