Problems with contenttypes when loading a fixture in Django

Question

I am having trouble loading Django fixtures into my MySQL database because of contenttypes conflicts. First I tried dumping the data from only my app like this:

./manage.py dumpdata escola > fixture.json

but I kept getting missing foreign key problems, because my app "escola" uses tables from other applications. I kept adding additional apps until I got to this:

./manage.py dumpdata contenttypes auth escola > fixture.json

Now the problem is the following constraint violation when I try to load the data as a test fixture:

IntegrityError: (1062, "Duplicate entry 'escola-t23aluno' for key 2")

It seems the problem is that Django is trying to dynamically recreate contenttypes with different primary key values that conflict with the primary key values from the fixture. This appears to be the same as bug documented here: http://code.djangoproject.com/ticket/7052

The problem is that the recommended workaround is to dump the contenttypes app which I'm already doing!? What gives? If it makes any difference I do have some custom model permissions as documented here: http://docs.djangoproject.com/en/dev/ref/models/options/#permissions

Solution 1

manage.py dumpdata --natural will use a more durable representation of foreign keys. In django they are called "natural keys". For example:

• Permission.codename is used in favour of Permission.id
• User.username is used in favour of User.id

Read more: natural keys section in "serializing django objects"

Some other useful arguments for dumpdata:

• --indent=4 make it human readable.
• -e sessions exclude session data
• -e admin exclude history of admin actions on admin site
• -e contenttypes -e auth.Permission exclude objects which are recreated automatically from schema every time during syncdb. Only use it together with --natural or else you might end up with badly aligned id numbers.

Solution 2

The answers here all old... As of 2017, the best answer is:

manage.py dumpdata --natural-foreign --natural-primary -e contenttypes -e auth.Permission --indent 4

Solution 3

Yes, this is really irritating. For a while I worked around it by doing a "manage.py reset" on the contenttypes app prior to loading the fixture (to get rid of the automatically-generated contenttypes data that differed from the dumped version). That worked, but eventually I got sick of the hassles and abandoned fixtures entirely in favor of straight SQL dumps (of course, then you lose DB portability).

update - the best answer is to use the --natural flag to dumpdata, as noted in an answer below. That flag did not exist yet when I wrote this answer.

Solution 4

Try skipping contenttypes when creating fixture:

./manage.py dumpdata --exclude contenttypes > fixture.json

It worked for me in a similar situation for unit tests, your insight regarding the contenttypes really helped!

Solution 5

I was not using MySQL but instead importing some data from a live server into sqlite. Clearing the contenttypes app data before performing loaddata did the trick:

from django.contrib.contenttypes.models import ContentType
ContentType.objects.all().delete()
quit()

And then

python manage.py loaddata data.json

View more solutions

Comments

[Beau]

I was running into this too, resetting the contenttypes app worked for me as well. Thanks for the tip!

[Oleg Tarasenko]

How did you reset them? In test case class? Give me an example please

[Carl Meyer]

I don't use fixtures for unittests, I generally create test data using the ORM in a setup() method because it's easier to keep in sync with the tests. So I never had to do this in a TestCase class, though I'm sure if you poke around in the code for Django's TestCase class you could figure out how to make a reset happen post syncdb and prior to fixture loading in a subclass. For me it was just "./manage.py reset contenttypes" in a bash script prior to "./manage.py loaddata my_fixture".

[philgo20]

I am using this to get natural keys for a content_type attribute, but I get this error when trying to load back the fixtures. TypeError: string indices must be integers, not str Any idea why ?

[Prince Paul]

For me, truncating the table prior to loaddata just causes different errors. No luck with this technique.

[wlnirvana]

@skyjur Why always use -e contenttypes -e auth.permission with --natural? I just tried without the --natural option and it worked. Also the documentation here says one should use this option if DUMPING auth.permission and contenttypes.

[Ski]

@winirvana because after you start from scratch and do syncdb, newly created ContentType and Permission are not guaranteed to get same id as they had before. Your data dump contains ids which might reference different objects on anather database where you will loaddata. It might worked for you because of one of these reasons: 1) your data didn't had any reference to these objects 2) original ids of Permission/ContentTypes were preserved 3) your loaddata was succesful but you actually have corrupt data due to objects referring to wrong objects and you don't yet know about it

[frnhr]

Flag --natural is now deprecated in favor of --natural-foreign (and --natural-primary)

[Paolo]

Final command could be: manage.py dumpdata --natural-foreign --natural-primary -e contenttypes -e auth.Permission --indent 4 > project_dump.json

[Code-Apprentice]

--natural has now been completely removed, not just deprecated. Use --natural-foreign or --natural-primary instead.

[yang zhou]

It will raise problem when loaddata, maybe mismatch with contenttype in new database

[Barney Szabolcs]

django.core.exceptions.ImproperlyConfigured: Requested setting INSTALLED_APPS, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.

[Barney Szabolcs]

It would probably work best within the handle of a custom management command.