Django Background Task

Question

I need to populate a SQLite database every few minutes in Django, but I want to serve stale data until the data is available for the database to be updated. (i.e. I don't want to block for the data to be gathered; the only time I can block is if there is a lock on the database, during which I have no choice.)

I also don't want to install a separate program or library.

How would I go about setting up another thread that could call save() on a bunch of models, without running into threading issues?

Solution 1

If you're looking for a lightweight solution for just executing stuff in background rather than a full-blown task management system, take a look at django-utils. It includes, among other things, an @async function decorator that will make a function execute asynchronously in a separate thread.

Use it like this:

from djutils.decorators import async

@async
def load_data_async():
    # this will be executed in a separate thread
    load_data()

Then you can call either the load_data_async function for background, or the normal load_data function for blocking execution.

Just make sure to install a version before 2.0, since that lacks the @async decorator.

Note: If even installing django-utils would be too much, you can simply download it and include the few required files in your project.

Solution 2

Celery.

Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well.

Celery is written in Python, but the protocol can be implemented in any language. It can also operate with other languages using webhooks.

Solution 3

Just a quick update on John Lehmann's answer: django-background-task was unmaintained and incompatible with newer Django version. We updated and extended it with new features a while ago and maintaining the new backward compatible package on Github. The new django-background-tasks app can be downloaded or installed from the PyPI.

Solution 4

Depends on whether you need the update to look atomic from the point of view of the readers. If you don't mind seeing old and new data together, just create a custom management command that populates the data, and run it every few minutes from cron.

If you need it to look atomic, wrapping the all the writes in one SQLite transaction via django.db.transaction should probably provide you with the necessary locks.

Solution 5

Django Background Task is a databased-backed work queue for Django, loosely based around Ruby's DelayedJob library.

You decorate functions to create tasks:

@background(schedule=60)
def notify_user(user_id):
    # lookup user by id and send them a message
    user = User.objects.get(pk=user_id)
    user.email_user('Here is a notification', 'You have been notified')

Though you still need something which schedules those tasks. Some benefits include automatic retries for failed tasks, and setting maximum duration for a running task.

This does involves another dependency but could be useful to some readers without that restriction.

View more solutions

Comments

[user541686]

I've never used cron before, but -- unless I'm misunderstanding this -- isn't that a completely separate program? Does that mean I have to create a separate, call cron from my server, and run that program to update the database?

[demux]

cron is a process that runs in background on linux, freebsd and osx. See: en.wikipedia.org/wiki/Cron.

[user541686]

@Amar: So like I'd previously understood, it's a completely separate program... which means I need to create a new program just for the update. I'm not sure I'm going to go that route, but +1, thanks for the idea anyway.

[che]

@Mehrdad: Almost every system has some sort of task scheduler. Unix-like ones have cron, Windows have en.wikipedia.org/wiki/Task_Scheduler

[Olli]

Unfortunately, djutils doesn't seem to be active anymore. At least readthedocs and github pages were removed.

[Don Kirkby]

The readthedocs pages still seem to be missing, @Olli, but the github pages are visible.

[PLNech]

The readthedocs link was erroneous : django-utils documentation does exist here.

[Jose Luis de la Rosa]

I tried using django-utils but it seems not compatible with Django 1.8, isn't it?

[Pablo]

I cannot do from djutils.decorators import async with Django 1.10 either. It wants to import hashcompat which has been deprecated in Django 1.6. I'd like to file a bug report, but the project seems to be dead :(