Atomic increment of a counter in django

Question

I'm trying to atomically increment a simple counter in Django. My code looks like this:

from models import Counter
from django.db import transaction

@transaction.commit_on_success
def increment_counter(name):
    counter = Counter.objects.get_or_create(name = name)[0]
    counter.count += 1
    counter.save()

If I understand Django correctly, this should wrap the function in a transaction and make the increment atomic. But it doesn't work and there is a race condition in the counter update. How can this code be made thread-safe?

Solution 1

Use an F expression:

from django.db.models import F

either in update():

Counter.objects.get_or_create(name=name)
Counter.objects.filter(name=name).update(count=F("count") + 1)

or on the object instance:

counter, _ = Counter.objects.get_or_create(name=name)
counter.count = F("count") + 1
counter.save(update_fields=["count"])

Remember to specify update_fields, or you might encounter race conditions on other fields of the model.

A note on the race condition avoided by using F expressions has been added to the official documentation.

Solution 2

In Django 1.4 there is support for SELECT ... FOR UPDATE clauses, using database locks to make sure no data is accesses concurrently by mistake.

Solution 3

If you don't need to know the value of the counter when you set it, the top answer is definitely your best bet:

counter, _ = Counter.objects.get_or_create(name = name)
counter.count = F('count') + 1
counter.save()

This tells your database to add 1 to the value of count, which it can do perfectly well without blocking other operations. The drawback is that you have no way of knowing what count you just set. If two threads simultaneously hit this function, they would both see the same value, and would both tell the db to add 1. The db would end up adding 2 as expected, but you won't know which one went first.

If you do care about the count right now, you can use the select_for_update option referenced by Emil Stenstrom. Here's what that looks like:

from models import Counter
from django.db import transaction

@transaction.atomic
def increment_counter(name):
    counter = (Counter.objects
               .select_for_update()
               .get_or_create(name=name)[0]
    counter.count += 1
    counter.save()

This reads the current value and locks matching rows until the end of the transaction. Now only one worker can read at a time. See the docs for more on select_for_update.

Solution 4

Keeping it simple and building on @Oduvan's answer:

counter, created = Counter.objects.get_or_create(name = name, 
                                                 defaults={'count':1})
if not created:
    counter.count = F('count') +1
    counter.save()

The advantage here is that if the object was created in the first statement, you don't have to do any further updates.

Solution 5

Django 1.7

from django.db.models import F

counter, created = Counter.objects.get_or_create(name = name)
counter.count = F('count') +1
counter.save()

View more solutions

Comments

[alexef]

should this be wrapped in a commit_on_success method?

[slacy]

The asker was specifically asking for how to atomically increment a field in the database.

[Bialecki]

One issue with this is if you need the updated value afterwards, you need to fetch it from the database. In certain cases, like ID generation, this can cause race conditions. For instance, two threads might increment an ID atomically (say from 1 to 3), but then both query for the current value and get 3, try to insert, explosion... Just something to think about.

[Bialecki]

This was the solution I ended up going with combined with wrapping the block in the transaction.commit_on_success.

[mlissner]

In the second version, why not use the defaults kwarg to get_or_create, and then put the F object inside a if created block? Should be faster in the case of creation, right? I went ahead and put an answer demoing what I mean.

[Han He]

This is definitely the right answer. Check the django doc about F(): Another benefit of using F() is that having the database - rather than Python - update a field’s value avoids a race condition.

[N. McA.]

In this guy's defence, that isn't stated explicitly. Clearly intended though.

[Alex Hall]

get_or_create returns a pair, so it should be counter, created = ... like in mlissner's answer.

[Alex Hall]

@Bialecki you could just write new_count = counter.count + 1 before doing the update.

[Aaron McMillin]

This answer has the best explanation. It wasn't till reading this one that I was convinced that count = F('count') + 1 would work

[Nandesh]

So, basically after updating a field like that,model will hold an instance of django.db.models.expressions.CombinedExpression, instead of the actual result. If you want to access the result immediately: "counter.refresh_from_db()"

[Whadupapp]

Every once in a while this fails for me and I end up with a smaller count than expected. Does every database backend support this? More specifically, does sqlite3 support this?

[George Y]

Exactly. The real problem is the race of the same data by different clients. An internal lock can make sure each client gets its own serial for its transaction.

[AnonymousUser]

I'm looking for a way to increment a field value, but on the page I don't want to need to refresh it, to get the new value when I click the increment button.