FreeBSD Listen Queue Overflows - can't increase max queue size

Question

I have a decently high trafficked FreeBSD Nginx server, and I'm starting to get a large number of listen queue overflows:

[root@svr ~]# netstat -sp tcp | fgrep listen
80361931 listen queue overflows
[root@svr ~]# netstat -Lan | grep "*.80"
tcp4  192/0/128        *.80
[root@svr ~]# sysctl kern.ipc.somaxconn
kern.ipc.somaxconn: 12288
[root@svr ~]#

However I can't seem to increase the max listen queue length past 128. I've increased kern.ipc.somaxconn, but it's not changing the max. Am I missing something?

Thanks!

Solution 1

May be listen queue limited to 128 in nginx config

Look nginx config for settings like:

listen 80 backlog=128;

And either remove backlog (default is -1 = use system limit) or change to bigger value (8192 should be enough even for loaded server). If even with increased listen queue your still see listen queue overflow it may indicate, that nginx is blocked for a long time (because of slow/overloaded HDD, or by bad written 3rd-party module).

Solution 2

The kern.ipc.somaxconn might not do what you think it does. It is the limit to outstanding and unhandled connections. (e.g.. it is not a connection limit, but the pending handle connection limit).

To use a non computer analogy: It is the maximum number of ringing phones (before they are picked up and answered), not the number of maximum simultaneous phone calls.

If you have backlog that is that big then your need to let your application pick up the phone more often (e.g. give it more resources, more CPU, better framework, whatever).

Note that the http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/configtuning-kernel-limits.html">FreeBSD handbook section on kernel limits states this: (Emphasis mine).

The kern.ipc.somaxconn sysctl variable limits the size of the listen queue for accepting new TCP connections. The default value of 128 is typically too low for robust handling of new connections in a heavily loaded web server environment. For such environments, it is recommended to increase this value to 1024 or higher. The service daemon may itself limit the listen queue size

I have no experience with Nginx, but also check its configuration files for above mentioned limit from the application side.

Comments

[Harry]

Thanks for the answer - I'm aware it's a queue, and during spikes it's too much for Nginx to handle, however there are sufficient resources to process it (even after a second or two delay, which is fine). I don't believe this is an Nginx problem because netstat is a system command; the OS is showing the queue overflows (OS level, not app level). My issue is as simple as how to increase the 128 max listen queue size... I believe that'll solve my problem, I just can't seem to get FreeBSD to increase it. Thanks!

[Hennes]

The OS is the part where the connection needs to be processed first (before it can hand it over to the application). So yes, if the apps can not keep up it will get dropped at OS level. If you increase OS buffers so it can buffer spikes then that should work. If Nginx is consistently slower then it is not going to work. But for brief spikes, aye, I think it should work.

[Philip]

Yes, changeing kern.ipc.somaxconn will increase the system limit. It will not affect any applications that are currently running. Applications would have to close listening sockets and re-open them at a minimum to change their limit. Nginx documentation say their limit is "-1" by default, which tells the OS to use the system limit. I haven't dug through any source code or bug reports to verify, but it sounds reasonable.

[Harry]

I'm aware of that. Shouldn't netstat be reporting whatever the system limit is? Look at the output of my original post - I've increased somaxconn, but netcat is still reporting the system limit as 128. Again, this was my original question/issue. Thanks!

[Philip]

If sysctl kern.ipc.somaxconn reports more than X, you restart nginx, and netstat is still showing X then nginx is requesting the "lower than the system limit" number. Looks like it would normally be configured as listen *:80 backlog=X in nginx's configuration. This is a "normal" server, correct? Not a VPS or shared hosting environment?

[Harry]

Yes, it's a normal, dedicated server. And there is nothing special in Nginx's configuration - vanilla with a few basic values raised. I have tried restarting Nginx to no avail. However I still don't quite understand how this could have anything to do with Nginx at all? I'm trying to increase the OS listen buffer queue... netstat should be reading the system queue... completely independent of any software running on the server. I'm just trying to increase the system max, but somaxconn isn't working...

[SaveTheRbtz]

Also, did you restart nginx after changing kern.ipc.somaxconn?