In celery 3.1, making django periodic task

Question

Things changed too much in Django, so I can't use 3.1. I need some help.

I read about make a task in django, and read Periodic Tasks document. But I don't know how make periodic tasks in django. I think this becuase of my low level English..

In the older version of Celery, I imported djcelery&crontab and set CELERYBEAT_SCHEDULE in settings.py, and excuted by manage.py.

But it seems that I cannot execute celery deamon by that way anymore. Than where I should put CELERYBEAT_SCHEDULE? In django example in docs, they set os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'proj.settings') in proj/proj/celery.py. (1) So setting celerybeat in settings.py(like before) is ok?

(2) If you can, a tiny example of making periodic task in django will very helpful. When I was reading tutorials, the most confusing part was File Path. If you don't want to provide a whole example, I will really appreciate if you explain about where I should make tasks, set beat, and executed deamon.

Thanks for reading.

Solution 1

I assume you've already read the django section from the docs, but have you seen this example project?

It doesn't use the scheduler but if you add this to settings.py:

from __future__ import absolute_import

from celery.schedules import crontab


CELERYBEAT_SCHEDULE = {
    # crontab(hour=0, minute=0, day_of_week='saturday')
    'schedule-name': {  # example: 'file-backup' 
        'task': 'some_django_app.tasks....',  # example: 'files.tasks.cleanup' 
        'schedule': crontab(...)
    },
}

# if you want to place the schedule file relative to your project or something:
CELERYBEAT_SCHEDULE_FILENAME = "some/path/and/filename"

Now for the commands, forget about manage.py, just type celery directly:

-B enables celery beat as always.

-A specifies the name of the celery app. Note this line in the celery.py of the example project: app = Celery('proj')

celery -A proj worker -B -l info

'django-celery' is not required, install it ONLY if you need to manage the schedule from the admin, or if you want to store task results in the DB through django's ORM:

INSTALLED_APPS += ('djcelery',)

# store schedule in the DB:
CELERYBEAT_SCHEDULER = 'djcelery.schedulers.DatabaseScheduler'

Solution 2

You can use django-celery application: https://pypi.python.org/pypi/django-celery

Installation:

pip install django-celery

To enable django-celery for your project you need to add djcelery to INSTALLED_APPS:

INSTALLED_APPS += ("djcelery", )
CELERYBEAT_SCHEDULER = "djcelery.schedulers.DatabaseScheduler"

then add the following lines to your settings.py:

import djcelery
djcelery.setup_loader()

USAGE

On linux you can run worker with celery-beat like this:

python manage.py celeryd worker --loglevel=DEBUG  -E -B -c 1
python manage.py help celeryd #to find out the args meaning

Also you will like to monitor tasks in django admin. To enable monitoring fiture you'll need to run celerycam:

python /var/www/gorod/manage.py celerycam

To make periodic task you can use celery.decorators.periodic_task.

# myapp/tasks.py
import datetime
import celery

@celery.decorators.periodic_task(run_every=datetime.timedelta(minutes=5))
def myfunc():
    print 'periodic_task'

Or use

# settings.py
CELERYBEAT_SCHEDULE = {
    'add-every-30-seconds': {
        'task': 'tasks.add',
        'schedule': timedelta(seconds=30),
        'args': (16, 16)
    },
}

Comments

[tadasajon]

from celery.schedules import crontab This line gives me the error: "No module named schedules"

[Adrián]

@JonCrowell just checked and the import is correct. You must have something wrong in your setup.

[xj9]

Did you add from __future__ import absolute_import do your settings file? If not python will try to import schedules from your celery.py file.

[noooooooob]

Straight from celery doc """You can also start embed beat inside the worker by enabling workers -B option, this is convenient if you will never run more than one worker node, but it’s not commonly used and for that reason is NOT RECOMMENDED for production use: $ celery -A proj worker -B"""

[Mark Chackerian]

@asksol if you are out there, please add from __future__ import absolute_import to the example @ celery.readthedocs.org/en/latest/userguide/periodic-tasks.ht‌​ml

[StockB]

The @periodic_task decorator is deprecated in Celery 3.1. github.com/celery/celery/issues/1764

[HorseHair]

Note - now this needs to be called CELERY_BEAT_SCHEDULE (recent versions)