"nf_conntrack: table full, dropping packet" even though nf_conntrack_count is much less than nf_conntrack_max

Question

I have a node in our cluster which gets lots of "nf_conntrack: table full, dropping packet" messages in the syslog. I checked the nf_conntrack_count and it was running right up against the nf_conntrack_max. Looking into the table, I saw most of the entries were DNS requests, so I added these rules to the "raw" netfilter table.

$ sudo iptables -t raw -vnL
Chain PREROUTING (policy ACCEPT 146M packets, 19G bytes)
pkts bytes target     prot opt in     out     source               destination
33M 4144M CT         udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp spt:53 CT notrack
33M 2805M CT         udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:53 CT notrack
Chain OUTPUT (policy ACCEPT 73M packets, 8311M bytes)
pkts bytes target     prot opt in     out     source               destination         
10785  882K CT         udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:53 CT notrack
0     0 CT         udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp spt:53 CT notrack

That left the count hovering around 13000 or so, and nf_conntrack_max is set to 65535. However I still keep getting the dropped packet messages. Most of the rest of the packets are UDP, and I set the nf_conntrack_udp_timeout as low as 1 second, leaving the nf_conntrack_count around 1000. However I still get the dropped packed message.

From here, if I raise the max, it will stop the dropped packet messages, however I don't see why that is necessary.

I am running docker, and there is an elasticsearch container (this problem seems to happen on whichever node is running elasticsearch). Not sure if it is relevant, but the node has 48 cores.

$ uname -a
Linux qtausc-pphd0128 3.19.0-26-generic #28~14.04.1-Ubuntu SMP Wed Aug 12 14:09:17 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

So why is it dropping packets when the count is far less than the max?

Solution

I had the same issue a while back on a Squid system.

One of the most effective way I found to reduce the size of the conntrack was to reduce the default TCP timeout in the kernel.

The net.netfilter.nf_conntrack_tcp_timeout_established is set to 432000 by default. That's right...that's 5 days.

To set the value you can issue the following command;

sysctl -w net.netfilter.nf_conntrack_tcp_timeout_established=X

And if you want that change to be persistent you need to add the line to /etc/sysctl.conf.

After reducing that value to 600 the conntrack count was steadily going down over a couple of days.

I used sysctl net.netfilter.nf_conntrack_max and sysctl net.netfilter.nf_conntrack_count in order to get the values.

Comments

[Matthew Sharp]

I was getting the count using cat /proc/sys/net/netfilter/nf_conntrack_count, which I believe is equivalent to sysctl net.netfilter.nf_conntrack_count except for output formatting. The thing is I was setting my UDP timeout really low (most of the traffic on my box in UDP, not TCP like your case), and I could get the conntrack_count down really low; much lower than the max. However I was still getting "packed dropped" messages, which is what is confusing me.