Monitoring Celery, what should I use?

Question

I'm using Django, Celery, and Django-Celery. I'd like to monitor the state/results of my tasks, but I'm a little confused on how to do that.

Do I use ./manage.py celeryev, ./manage.py celerymon, ./manage.py celerycam? Do I run sudo /etc/init.d/celeryevcam start?

Solution 1

Run:

./manage.py celeryd -E
./manage.py celerycam

The first starts a worker with events enabled.

Now you can find task results in the django admin interface. Ensure you have djcelery in your INSTALLED_APPS. If you want to see task state use also celerymon.

Solution 2

I know this is an old question. But for those visiting in the future, Celery Flower is a great tool. It is also recommended by Celery. Here are some cool features from the link above:

Real-time monitoring using Celery Events

• Task progress and history
• Ability to show task details (arguments, start time, runtime, and more)
• Graphs and statistics

Remote Control

• View worker status and statistics
• Shutdown and restart worker instances
• Control worker pool size and autoscale settings
• View and modify the queues a worker instance consumes from
• View currently running tasks
• View scheduled tasks (ETA/countdown)
• View reserved and revoked tasks
• Apply time and rate limits
• Configuration viewer
• Revoke or terminate tasks

Broker monitoring

• View statistics for all Celery queues
• Queue length graphs

HTTP API

Basic Auth, GitHub OAuth2 and Google OpenID authentication

Comments

[Seán Hayes]

Thanks! So, what does the celeryevcam init script do? I looked at its source and it looks like it runs celeryev, which I thought was an curses event viewer, which wouldn't make sense to run as a daemon with no one to look at it, right?

[Seán Hayes]

Wait, I think I've got it. celeryev is normally used for celery on its own and doubles as a viewer and camera, but django-celery provides celerycam which just does the camera part, and the init script is called celeryevcam because you can use it with either celeryev or celerycam. Is that right?

[vad]

i never heard celeryevcam, sorry

[Seán Hayes]

It's one of the daemon scripts: github.com/ask/celery/blob/master/contrib/debian/init.d/…

[cerberos]

The link has changed to github.com/ask/celery/blob/master/contrib/generic-init.d/…, this can be used to take the snapshots.

[ostrokach]

As of version 3.1, celeryd has been superseded by celery worker, so the first command should be ./manage.py worker -E. Here is a comparison of old and new celery commands.