nginx proxy_cache: limit parallel requests to a backend

Question

I am using nginx as a reverse proxy to my backend(s). Configuration is pretty basic, e.g. the core is just:

upstream myservice {
    server 127.0.0.1:80;
    server 123.123.123.123:80;
}

location / {
    proxy_pass  http://myservice;
    proxy_set_header Host myservice;
}

Now my service is computationally very heavy, and I would like nginx to limit the number of active parallel (simultaneous) requests to a single upstream backend to e.g. 10.

I looked into limit_req module, however this module only seems to focus on incoming requests per minute. I specifically would like to limit the number of active backend connections; i.e. take into account if requests have already returned or not. Is this possible at all?

In Varnish this can be done using e.g.

backend cpu1 {
  .host = "127.0.0.1";
  .port = "80";
  .max_connections = 20;
}

However I need to use nginx for this.

Solution 1

Take a look at the limit_conn module. While all the examples I've found have been on limiting by remote IP, if you pass a constant to limit_conn_zone instead of the remote IP you will be limiting by total connections to the server.

Something like this:

upstream myservice {
    server 127.0.0.1:80;
    server 123.123.123.123:80;
}

set $const 1;
limit_conn_zone $const zone=one:1m;

location / {
    limit_conn one 10;
    proxy_pass  http://myservice;
    proxy_set_header Host myservice;
}

Edit: It seems like this module is only available in more recent versions of nginx (possibly >=v1.1.8). You may need to compile from source if you want to use this option and have an older version of nginx.

Edit 2: If your nginx is only doing proxy duties, take a look at worker_connections

Solution 2

I think the configuration options you require are:

proxy_cache_use_stale updating;
proxy_cache_lock on;

This serves out of date elements from cache, and issuing updates from backend. So when your backend servers a page after 2 seconds, the client already got the old ressource from cache.

The second locks one element from cache, while already a update-cache request to backend is running, so there are no paralell requests for one cache-item.

This should be much better then hard proxy limits.

If you still want hard proxy limits for connections, you can run a nginx in backend and rate-limit connections or requests there. Else you might want to try a connection limit with iptables, which should perform much better than in nginx.

Comments

[Jeroen Ooms]

Thanks, I hadn't found this module yet. It seems that Ubuntu 12.04 will ship with nginx 1.1.19-1 so that's nice.

[Jeroen Ooms]

Do you think limit_conn could also appear in the upstream {} directive? I would really like to limit the upstream lookup so that when caching is enabled in the proxy, cache hits do not count to the limit.

[dwurf]

I think the important thing here is your choice of variable in the limit_conn_zone statement. If you can figure out a variable that is unique for each cache hit, but constant for each proxy pass-through, then you will not be counting cache hits towards the limit.