Django stops working with RuntimeError: populate() isn't reentrant

Question

I've been developing a Django web application deployed on an Apache server with WSGI, and everything has been going smoothly. Today, I made some minor changes to my app's admin.py in an attempt to customize the build-in Django Admin interface, and initially made a syntax error (an unclosed parenthesis). This meant that when I touched wsgi.py and loaded the code (I have WSGI running in daemon mode on my virtual host), my website was replaced with an Internal Server Error because WSGI stopped when it hit the syntax error.

So I fixed the syntax error, checked that I didn't have any more with manage.py check, and touched wsgi.py to redeploy. But my website still displays an Internal Server Error! Checking the Apache logs, this is what I see:

[Sun Nov 23 13:52:46 2014] [info] mod_wsgi (pid=19093): Create interpreter 'quotes.cs.cornell.edu|'.
[Sun Nov 23 13:52:46 2014] [info] mod_wsgi (pid=19093): Adding '/extra/www/html/quotes/quotes_django' to path.
[Sun Nov 23 13:52:46 2014] [info] mod_wsgi (pid=19093): Adding '/opt/rh/python27/root/usr/lib64/python2.7/site-
packages/' to path.
[Sun Nov 23 13:52:46 2014] [info] [client 128.84.33.19] mod_wsgi (pid=19093, process='quotes.cs.cornell.edu',
  application='quotes.cs.cornell.edu|'): Loading WSGI script '/extra/www/html/quotes/quotes_django/quotes_django/
wsgi.py'.
[Sun Nov 23 13:52:46 2014] [error] [client 128.84.33.19] mod_wsgi (pid=19093): Target WSGI script '/extra/www/html/
quotes/quotes_django/quotes_django/wsgi.py' cannot be loaded as Python module.
[Sun Nov 23 13:52:46 2014] [error] [client 128.84.33.19] mod_wsgi (pid=19093): Exception occurred processing WSGI
script '/extra/www/html/quotes/quotes_django/quotes_django/wsgi.py'.
[Sun Nov 23 13:52:46 2014] [error] [client 128.84.33.19] Traceback (most recent call last):
[Sun Nov 23 13:52:46 2014] [error] [client 128.84.33.19]   File "/extra/www/html/quotes/quotes_django/
quotes_django/wsgi.py", line 14, in <module>
[Sun Nov 23 13:52:46 2014] [error] [client 128.84.33.19]     application = get_wsgi_application()
[Sun Nov 23 13:52:46 2014] [error] [client 128.84.33.19]   File "/opt/rh/python27/root/usr/lib64/python2.7/site-
packages/django/core/wsgi.py", line 14, in get_wsgi_application
[Sun Nov 23 13:52:46 2014] [error] [client 128.84.33.19]     django.setup()
[Sun Nov 23 13:52:46 2014] [error] [client 128.84.33.19]   File "/opt/rh/python27/root/usr/lib64/python2.7/site-
packages/django/__init__.py", line 21, in setup
[Sun Nov 23 13:52:46 2014] [error] [client 128.84.33.19]     apps.populate(settings.INSTALLED_APPS)
[Sun Nov 23 13:52:46 2014] [error] [client 128.84.33.19]   File "/opt/rh/python27/root/usr/lib64/python2.7/site-
packages/django/apps/registry.py", line 115, in populate
[Sun Nov 23 13:52:46 2014] [error] [client 128.84.33.19]     app_config.ready()
[Sun Nov 23 13:52:46 2014] [error] [client 128.84.33.19]   File "/opt/rh/python27/root/usr/lib64/python2.7/site-
packages/django/contrib/admin/apps.py", line 22, in ready
[Sun Nov 23 13:52:46 2014] [error] [client 128.84.33.19]     self.module.autodiscover()
[Sun Nov 23 13:52:46 2014] [error] [client 128.84.33.19]   File "/opt/rh/python27/root/usr/lib64/python2.7/site-
packages/django/contrib/admin/__init__.py", line 23, in autodiscover
[Sun Nov 23 13:52:46 2014] [error] [client 128.84.33.19]     autodiscover_modules('admin', register_to=site)
[Sun Nov 23 13:52:46 2014] [error] [client 128.84.33.19]   File "/opt/rh/python27/root/usr/lib64/python2.7/site-
packages/django/utils/module_loading.py", line 74, in autodiscover_modules
[Sun Nov 23 13:52:46 2014] [error] [client 128.84.33.19]     import_module('%s.%s' % (app_config.name,         
module_to_search))
[Sun Nov 23 13:52:46 2014] [error] [client 128.84.33.19]   File "/usr/lib64/python2.7/importlib/__init__.py", line 
37, in import_module
[Sun Nov 23 13:52:46 2014] [error] [client 128.84.33.19]     __import__(name)
[Sun Nov 23 13:52:46 2014] [error] [client 128.84.33.19]   File "/extra/www/html/quotes/quotes_django/quotespage/
admin.py", line 25
[Sun Nov 23 13:52:46 2014] [error] [client 128.84.33.19]     approve_quotes.short_description = "Approve selected
quotes"
[Sun Nov 23 13:52:46 2014] [error] [client 128.84.33.19]                  ^
[Sun Nov 23 13:52:46 2014] [error] [client 128.84.33.19] SyntaxError: invalid syntax
[Sun Nov 23 13:53:36 2014] [info] [client 128.84.33.19] mod_wsgi (pid=19093, process='quotes.cs.cornell.edu',
  application='quotes.cs.cornell.edu|'): Loading WSGI script '/extra/www/html/quotes/quotes_django/quotes_django/
wsgi.py'.
[Sun Nov 23 13:53:36 2014] [error] [client 128.84.33.19] mod_wsgi (pid=19093): Target WSGI script '/extra/www/html/
quotes/quotes_django/quotes_django/wsgi.py' cannot be loaded as Python module.
[Sun Nov 23 13:53:36 2014] [error] [client 128.84.33.19] mod_wsgi (pid=19093): Exception occurred processing WSGI
script '/extra/www/html/quotes/quotes_django/quotes_django/wsgi.py'.
[Sun Nov 23 13:53:36 2014] [error] [client 128.84.33.19] Traceback (most recent call last):
[Sun Nov 23 13:53:36 2014] [error] [client 128.84.33.19]   File "/extra/www/html/quotes/quotes_django/         
quotes_django/wsgi.py", line 14, in <module>
[Sun Nov 23 13:53:36 2014] [error] [client 128.84.33.19]     application = get_wsgi_application()
[Sun Nov 23 13:53:36 2014] [error] [client 128.84.33.19]   File "/opt/rh/python27/root/usr/lib64/python2.7/site-
packages/django/core/wsgi.py", line 14, in get_wsgi_application
[Sun Nov 23 13:53:36 2014] [error] [client 128.84.33.19]     django.setup()
[Sun Nov 23 13:53:36 2014] [error] [client 128.84.33.19]   File "/opt/rh/python27/root/usr/lib64/python2.7/site-
packages/django/__init__.py", line 21, in setup
[Sun Nov 23 13:53:36 2014] [error] [client 128.84.33.19]     apps.populate(settings.INSTALLED_APPS)
[Sun Nov 23 13:53:36 2014] [error] [client 128.84.33.19]   File "/opt/rh/python27/root/usr/lib64/python2.7/site-
packages/django/apps/registry.py", line 78, in populate
[Sun Nov 23 13:53:36 2014] [error] [client 128.84.33.19]     raise RuntimeError("populate() isn't reentrant")
[Sun Nov 23 13:53:36 2014] [error] [client 128.84.33.19] RuntimeError: populate() isn't reentrant

The first series of errors shows WSGI failing due to the syntax error in my admin.py. However, the second series of errors seems to show an error internal to Django:

RuntimeError: populate() isn't reentrant

thrown from the populate method of registry.py.

Googling this error message returns surprisingly little information, none of it from Django documentation. Apparently, it can sometimes happen if you name an app twice in your settings.py, but I'm not doing that. More importantly, I haven't changed settings.py since the point where the website was working fine -- the only thing I changed was admin.py.

I tried reverting all the changes I made, so all my Python code is back in the state it was when the website was working -- and I still get the populate() isn't reentrant error when I try to make WSGI reload the code!

I've also tried commenting-out different apps in the INSTALLED_APPS section of settings.py, and even with only 'django.contrib.staticfiles' enabled the error still happens. Weirdly, I still get the error even if I comment out all the apps -- Django throws the error even when it isn't loading any apps!

Does anyone know what's going on here? Or any better way for me to debug this error, since the traceback in the Apache log is pretty unhelpful?

Notes: I'm using Django 1.7, Apache 2.2, and Python 2.7.

Solution 1

This is caused by a bug in your Django settings somewhere. Unfortunately, Django's hiding the bug behind this generic and un-useful error message.

To reveal the true problem, open django/apps/registry.py and around line 80, replace:

raise RuntimeError("populate() isn't reentrant")

with:

self.app_configs = {}

This will allow Django to continue loading, and reveal the actual error.

I've encountered this error for several different causes. Once was because I had a bad import in one of my app's admin.py.

Solution 2

My server's administrator restarted Apache, and that magically fixed this problem. The exact same Python files loaded without causing populate() isn't reentrant. I even tried loading another file with a syntax error, then fixing it, and the server was able to load the new file and run correctly with no problems.

I still don't know what was going wrong, but I'm marking this as answered since the problem is gone. (Well, I'll mark it as answered as soon as StackOverflow allows me to accept my own answer.)

Update: After continuing to get this error when I accidentally upload Python with syntax errors, I figured out a workaround that's easier than restarting Apache. When WSGI starts throwing the populate() isn't reentrant error, I replace my Django project's wsgi.py with this simple function:

def application(environ, start_response):
    if environ['mod_wsgi.process_group'] != '': 
        import signal
        os.kill(os.getpid(), signal.SIGINT)
    return ["killed"]

Then I reload my website, and the WSGI daemon process restarts (which I can tell by looking at the Apache log, even though the website still displays the same 500 error).

If I then change wsgi.py back to normal and reload again, WSGI successfully picks up my code without throwing populate() isn't reentrant (assuming I have no syntax errors this time). So the entirety of Apache doesn't need to restart, just the WSGI process, and I can do that without root privileges.

Solution 3

I know this is an old answer but I will contribute with my solution:

As a way to diagnose the source of the problem run manage.py checkand see if you find anything there

In my case an outdated requirement was the issue and django was failing to import a submodule

Make sure that your requirements are up to date

Solution 4

It's not a response but a reflexion.

When you upgrade to django 1.7 and you have a 500 error and reload your page, Apache says "populate() isn't reentrant". I think it's when you load your page, Apache load all the modules you need for your app and when the error is handle it doesn't unload module. So, when you reload your page, apache load again theses modules but it's already loaded. So, apache says "populate() isn't reentrant".

I've two actions to correct this : Restart apache, or correct the error that handle the first 5OO error.

Try restarting apache with:

sudo service httpd restart

I hope it will help you.

Solution 5

If you're getting this error when using Google App Engine check your logs for other errors which might be causing this. I was getting:

ImproperlyConfigured: Error loading either pysqlite2 or sqlite3 modules (tried in that order): No module named _sqlite3

You can't use SQLite with Google App Engine so commenting out the DATABASES section of settings.py stopped that error and the RuntimeError("populate() isn't reentrant") error as well.

View more solutions

Comments

[dukebody]

I'd try removing all .pyc files that might be around.

[Edward]

Nope, removing all .pyc files didn't help. Touching wsgi.py results in the same Apache error, and the .pyc files aren't recreated.

[dukebody]

Have you tried restarting Apache?

[Edward]

I can't restart Apache because I don't have the rights to on this server. The administrator with root access won't be back in the office until Monday.

[maciek]

In my case it was not installed app from INSTALLED_APPS in environment.

[ggdx]

I know it's 4 years on but there are a LOT of different answers here. Can you @Edward recall if any solved the problem?

[Edward]

The only two answers that solved the problem for me are the one I posted (making wsgi.py kill itself) and the one @seddonym posted (touching an "earlier" file). However, it seems like this error can be caused by more than one kind of problem, so maybe the other answers have worked for other people in different situations.

[User]

In my case I had upgraded to Django 3 in my requirements file but didn't update psycopg2. Running pip install --upgrade psycopg2 fixed the problem

[user2662692]

For me the problem was not migrating changes I had made to a model in one of my apps.

[Undo]

You've posted the same answer to three questions. Please tailor each answer to the question.

[Rohit]

Ok, I am facing almost the same problem and yes it gets fixed on rebooting the machine and restarting the server. But the problem keeps repeating. Sometimes all of a sudden. within a month it has occured thrice. If anybody has any idea, please help.

[Edward]

@Rohit, if you want help with your situation, you should probably post a new question describing it. You're unlikely to get an answer in the comments here.

[Graham Dumpleton]

Create a new question. Don't ask question as answers.

[CoderGuy123]

I'm not asking a question. Posting a solution to the same problem that worked for me. Apache throws error 500 and the error in the log file is the same as the one in the question asked here. Please reread my answer. :)

[Graham Dumpleton]

Then include the actual solution in your answer rather than link out to another question. The problem is though that that other post doesn't mention the populate() issue at all and to me looks like an entirely different problem, so I can't see how you think it was solving the same issue. So as far as anyone can tell, you had a different problem which you should have asked a separate question for at the outset rather than confusing the responses here for what looks like a different problem based on details in that other post..

[Graham Dumpleton]

There is a startup-timeout option for daemon mode of mod_wsgi in more recent versions of mod_wsgi which assists in recovering from transient errors when Django is being initialised, such as a database not being available. The timeout will cause the process to be automatically restarted if WSGI application doesn't load properly after the timeout period. Even that will not help where you have a permanent issue with your own code. In that case, look for the very first error, not the populate() error as it will give the real reason your code fails.

[CoderGuy123]

It's unnecessary to repeat the steps here. The errors appear in the same log, just that OP here didn't include all of it. One error is shown when the page is accessed (that's the part in OP here), another when Apache starts (that's the part in the other thread).

[Ali Hesari]

I get Error 500 on Apache and I can't found the problem. By this command manage.py check I found the problem. Thank you.

[Cerin]

What do you mean "Django project's wsgi.py with this simple function"? How do you replace a file with a function?

[Edward]

@Cerin: That means the contents of the file became equal to the function I listed. I replaced a Python file with a different Python file, and the new Python file contained only one function.

[Golden Thumb]

This explains why everything worked fine locally but got 500 error when deployed to GAE. A small hint: searching text "raise" on GAE log page generated exact one hit which led to the error message pointed out by donturner. His solution saved my day :-)

[Graham Dumpleton]

More importantly, ensure you go back in the error log and look for the very first error message that occurs after a restart of one of the Apache or mod_wsgi daemon processes. That first error will tell you the true reason for the error, any requests after that for the same process will mention the reentrant error only. In daemon mode, also ensure you are using startup-timeout option if you have this problem due to transient errors, rather than coding errors. At least that way it can automatically recover.

[Babken Vardanyan]

Why the downvotes? This was the correct answer for me.

[ming]

but the tutorial says INSTALLED_APPS = [ 'allauth', 'allauth.account', 'allauth.socialaccount',]

[Swati Srivastava]

@ming It's optional.

[Jon]

We had the same problem with CentOS 7 and SELinux. We fixed it by using chcon to change the context of the problem .so file to httpd_sys_script_exec_t .

[Popieluch]

This is the right answer, the error is generic, check the message just before the runtimeerror.

[RickyA]

This is by far the most useful answer in this thread. Now I know -> django.core.exceptions.ImproperlyConfigured: psycopg2_version 2.5.4 or newer is required; you have 2.5 (dt dec pq3 ext)

[Qi Fan]

Looks like this error in general is caused by some long running Django service (such as wsgi daemon process) generating an import error. The first occurrence generates a correct error log including the import error but all subsequent web access would generate this "reentrant" error. It seems that Django tries to reload the failed apps but they are "already imported" for the reentrant check.

[Ángel Jiménez]

Saved my life. This solution allowed me to check successfully on the missing requirements and compatibility errors between them.

[luislhl]

Is this dangerous to try in a production environment? I'm getting those erros in production, but don't know how to reproduce them in my machine.

[James]

This was perfect. Changing that one line let me see I had a simple import error. I couldn't tell what was going on before that.

[xpanta]

This answer is pure gold.

[User]

@RickyA and this was by far the most useful comment as I didn't even have to bother with this answer and just tried pip install --upgrade psycopg2 and that fixed it.

[djvg]

This occurs e.g. if you deploy to AWS EB using Django version 2.2 (or later) with the default sqlite database. You'll get django.core.exceptions.ImproperlyConfigured: SQLite 3.8.3 or later is required (found 3.7.17). followed by the RuntimeError: populate() isn't reentrant. From the AWS docs: "Django 2.2 is incompatible with the Elastic Beanstalk Python 3.6 platform." (at the time of writing)

[babis21]

Well, this accepted solution might have worked for you, but I wouldn't regard it as accepted, as this is quite an ugly fix. To those that may face this problem, for me the reason was an error in the python code (in more detail, I upgraded django and a third party package was breaking and had to update this package as well. But this error went unnoticed and was raised during app initialization which then caused apps to reload, hence the exception). I also had this in the past, and again there was some error during the load process. So debugging your app upon initialize is a good place to start.

[Edward]

There are actually two different problems here, and two different kinds of solutions. One problem is that there is an error in your Python code that makes your app fail to initialize correctly, and then as @Cerin's answer points out, Django "hides" the real problem behind the unhelpful "populate isn't re-entrant" message. The other problem is that, even once you fix the error in your code, Django will still keep failing with this message because WSGI won't reload your fixed code until you forcibly kill it. My solution fixes the latter problem.

[Ori]

It should definitely be the right answer, and furthermore this should be updated in Django :) Thanks!

[mhellmeier]

If someone uses django with docker (or docker-compose), the line can easily be replaced in the Dockerfile while creating the image by adding this line: RUN sed -i "s/raise RuntimeError(\"populate() isn't reentrant\")/self.app_configs = {}/g" /usr/local/lib/python3.7/site-packages/django/apps/registry.‌​py (the Python version may have to be adapted).

[Maksym]

This required removing cached compiled registry.pyc and restarting httpd. But it did finally tell me what's wrong!

[Eerik Sven Puudist]

Reloading apache worked for me as well, but I had to type sudo /etc/init.d/apache2 reload

[Menas]

I'm very impressed by how much in-depth knowledge of the internals of Django library files this answer demonstrates!!

[DjangoDev1]

Is there another option for this? I'm using some Celery tasks and would like access to the models so If I add django.setup(), the tasks works but I get Populate isn't reentrant.

[ThorSummoner]

are you using django-celery specifically? (docs.celeryproject.org/en/latest/django/…) I think part of the move to the @shared_task decorator is to make celery init play nice with django init

[DjangoDev1]

I've been following a lot of tutorials and I only have one tasks.py file in which I keep all of the tasks. I've posted a new question about this. - stackoverflow.com/questions/66648110/…

[Cornflex]

Great, much prefer this over uncommenting lines of django code! Thanks!

[Cornflex]

As per this answer, python manage.py check achieves the same without the need to alter django code.

[danny]

I get Error 500 on IIS, by manage.py check command, found the problem. Some new packages missing. Thanks very much.

[Jeet Patel]

How do I navigate to django/apps/registry.py directory. I have installed django using pip.

[Phil]

using manage.py check actually raised RuntimeError: populate() isn't reentrant for me..

[David Rhoden]

How do you do that?

[Vimuth]

This was so useful, I logged in to my S/O account from my work laptop which involved me changing my google password, just so I can upvote this.

[Harley]

sudo systemctl restart apache2 now I've got other issues!