create multiple objects without multiple hits to the db in Django 1.8

Question

With Django 1.8 how do I create multiple objects and write to db in a single db transaction?

It seems previous versions had @commit_manually. But when I try to modify this code: https://stackoverflow.com/a/29834940 I get an exception

django.db.transaction.TransactionManagementError: The outermost 'atomic' block cannot use savepoint = False when autocommit is off.

At lot of posts I’ve seen say to wrap in “with transaction.atomic()” block but this hits the db in every loop.

Here is a simplified version of my code. I haven’t ran it but it should show what I’m trying to do.

class Foo(models.Model):
    Name = models.CharField( max_length=200 )

class Bar(models.Model):
    Foos = models.ManyToManyField( foo )

class Processor(object):
    def run(self,):
        myBar = Bar.object.create()

        myList = ['a', 'b', 'c', 'd']

        if True:
            set_autocommit( False )

            for char in myList:
                myFoo = Foo.objects.create(Name=char)
                myBar.Foos.add( myFoo )

            commit()
            set_autocommit( True )

I'm trying to make all these changes and only hit the db once. I know there is model.objects.bulk_create but I couldn't find a what to handle the m2m relationships. Also bulk_create doesn't return the the db instances so I have to pull them out of the db for the m2m relationships.

Solution

You can't do all of them in one query. However, you are on the right track. You should be using .bulk_create() for batch insertion and do one more lookup for getting them back as objects in order to add into foos.

If you are too concerned that the latter query will perform slowly, you can set unique=True or, db_index=True for improving performance.

You also want to maintain the atomicity of operations using transactions, so that any INSERT fails, all of them should be rolled back:

from django.db import transaction

class Foo(models.Model):
    name = models.CharField(max_length=200, unique=True)

class Bar(models.Model):
    foos = models.ManyToManyField(Foo)

class Processor(object):
    def run(self):
        transaction.set_autocommit(False)
        try:
            myList = ['a', 'b', 'c', 'd']
            Foo.objects.bulk_create([Foo(n) for n in myList])

            myBar = Bar.object.create()
            myBar.foos.add(Foo.objects.filter(name__in=myList))
        except:
            transaction.rollback()
            raise
        else:
            transaction.commit()
        finally:
            transaction.set_autocommit(True)

Please see Django documentation for more information about autocommit behavior.