Gunicorn with max-request limit blocks on high load

Question

I'm trying to understand the following scenario:

• I have a website with nginx in front (serving with SSL, config see below)
• requests to the Django application are handled by gunicorn (0.18, config see below, managed by supervisord)
• when a user loads the website, 10 requests are handled by the gunicorn (the other ones are static files served by nginx) - this requests are not long running requests
• the gunicorn is configured to take maximum of 1000 requests per worker until the worker is respawned
• about 450 people are able to load the page within a short time range (1-2 minutes)
• afterwards the gunicorn somehow blocks and does not handle any more connections, the result is that nginx responds with Gateway Timeout after a while

I suppose the restarting of the workers does not really happen or the mechanism is blocked by the load? I want to understand what is happening to fix this issue.

Can anyone explain what is happening here? Thanks a lot!

PS: I'm tied to use gunicorn 18.0, newer version is currently not possible.

Here are the configs I use.

nginx:

# nginx
upstream gunicorn_app {
    server 127.0.0.1:8100;
}
server {
    listen 443 ssl;
    ...
    # skipping static files config
    ...
    location @proxy_gunicorn_app {
        proxy_read_timeout 1800;
        proxy_set_header   Host $host;
        proxy_set_header   X-Real-IP $remote_addr;
        proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Proto https;
        proxy_pass         http://gunicorn_app;
    }
}

gunicorn (started via supervisord):

# gunicorn
python manage run_gunicorn --workers 4 --max-requests 1000 -b 127.0.0.1:8100 --timeout 1800

Solution

Not really sure of what might be the issue here.

But, you can try debugging using the Server hooks like:

• on_reload: Called to recycle workers during a reload via SIGHUP. The callable needs to accept a single instance variable for the Arbiter.

  def on_reload(server): #Print some debug message

• worker_int: Called just after a worker exited on SIGINT or SIGQUIT.

  def worker_int(worker): #Print some debug message

• pre_request: Called just before a worker processes the request.

  def pre_request(worker, req): #Print some debug message #worker.log.debug("%s %s" % (req.method, req.path))

• post_request: Called after a worker processes the request.

  def post_request(worker, req, environ, resp): #Print some debug message

This might help you reach the root of the problem.

Reference in the gunicorn docs: http://docs.gunicorn.org/en/stable/settings.html#server-hooks