Apache2 mod_wsgi python 2.6 Django very slow

Question

I've tried about 1000x things, but I can't seem to figure out why a simple django website is slow using apache 2.2.14/wsgi latest / django 1.3. I've confirmed the problem isn't our database by turning on mysql slow query logging. I've reviewed the wsgi daemon configuration settings about another 100x times but still don't understand why runserver is currently faster than apache!

Here is my apache config, let me know if there are other items that would be useful!

WSGIDaemonProcess xxx display-name=xxx group=www-data user=www-data processes=25 threads=1
<VirtualHost *:80>
    ServerName www.xxx.com
    ServerAlias xxx.com
    ServerAlias localhost
    ServerAdmin [email protected]

    RewriteEngine Off
        RewriteCond %{HTTP_HOST} ^xxx\.com$ [NC]
        RewriteRule ^(.*)$ http://www.xxx.com$1 [R=301,L]
        RewriteCond %{REQUEST_URI} ^/login/$
        RewriteRule /login https://%{HTTP_HOST}%{REQUEST_URI} [R,L]
        RewriteCond %{REQUEST_URI} ^/signup/
        RewriteRule /signup https://%{HTTP_HOST}%{REQUEST_URI} [R,L]

    ErrorLog /var/log/apache2/xxx-error.log
    LogLevel debug
    CustomLog /var/log/apache2/xxx-access.log combined

    WSGIProcessGroup %{GLOBAL}
    WSGIApplicationGroup %{GLOBAL}
    WSGIScriptAlias / /srv/xxx.com/mod_wsgi/dispatch.wsgi

Alias /static /srv/xxx.com/src/xxx/static
<Directory "/srv/xxx.com/src/xxx/static">
    Order deny,allow
    Allow from all
</Directory>
</VirtualHost>

free:

             total       used       free     shared    buffers     cached
Mem:           496        489          6          0          1         17
-/+ buffers/cache:        471         25
Swap:         1023         50        973

top:

top - 21:30:52 up  2:06,  1 user,  load average: 0.07, 0.10, 0.12
Tasks: 101 total,   2 running,  99 sleeping,   0 stopped,   0 zombie
Cpu(s):  1.2%us,  1.2%sy,  0.0%ni, 95.4%id,  2.2%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:    508272k total,   467788k used,    40484k free,     1448k buffers
Swap:  1048572k total,    59576k used,   988996k free,    22708k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                             
 5009 www-data  20   0  179m  41m 5024 R   20  8.3   0:02.59 apache2                                                              
 2521 elastic   20   0  710m  70m 4052 S    1 14.2   0:54.32 java                                                                 
 5013 root      20   0 19272 1252  932 R    0  0.2   0:00.05 top                                                                  
    1 root      20   0 23628 1108  784 S    0  0.2   0:00.18 init                                                                 
    2 root      20   0     0    0    0 S    0  0.0   0:00.00 kthreadd   

Here is the mpm_prefork_module settings:

<IfModule mpm_prefork_module>
    StartServers          5
    MinSpareServers       5
    MaxSpareServers      10
    MaxClients          150
    MaxRequestsPerChild   0
</IfModule>

Solution

You have:

WSGIDaemonProcess xxx display-name=xxx group=www-data user=www-data processes=25 threads=1

but then have:

WSGIProcessGroup %{GLOBAL}

which means that you aren't delegating the WSGI application to run in that daemon process group.

In other words, you are running your WSGI application in embedded mode instead and the WSGIDaemonProcess directive is redundant.

If you are also using Apache prefork MPM you are likely suffering possible speed issues due to Apache using up to 150 single threaded processes in its default configuration.

The slowness is thus likely due to the lazy loading of your WSGI application, if it is large, when a request actually comes in.

As more requests come in, Apache has to keep spinning up new processes to meet increased demand. If requests drop off, Apache will start dropping processes. If another increase in requests comes in, it has to start spinning up new processes again and loading your application again.

Now this is an extreme scenario, and how badly you may be hit by it depends on how the Apache MPM settings are set, which you don't show, and what your traffic profile is like.

In worst case you may even have overriden MaxRequestsPerChild directive and telling Apache to kill process after a single or a small number of requests and so you may be forcing reloads of your application all the time.

For some reading about related problems for this sort of issue read:

http://blog.dscpl.com.au/2009/03/load-spikes-and-excessive-memory-usage.html

So that is how things could go bad based on based Apache configuration.

Ignoring the wrong configuration for daemon mode, you have the possibility of it being an application issue. For that I would suggest trying a performance monitoring tool. My biased suggestion there is New Relic. Even if you don't want to pay for such a tool, it gives two weeks of trial for all features which could be enough for you to sort out where the bottleneck is.

For an example of what New Relic can do for Python, look at:

http://blog.newrelic.com/2011/11/08/new-relic-supports-python/

Comments

[Jonathan S. Fisher]

so if I set my WSGIProcessGroup to xxx what should I set WSGIApplicationGroup to?

[Graham Dumpleton]

Since you only have the one WSGI application, you can leave WSGIApplicationGroup as %{GLOBAL}. That directive says which sub interpreter within process to use. %{GLOBAL} is a special one which equates to main interpreter, ie., first one created. It is equivalent to what is used when command line interpreter is used. Some C extension modules for Python will only work properly when run in main interpreter, so is generally a good idea to force application to run in main interpreter when you can.