ip_conntrack_count has way too many connections

Question

The conntrack table on my server has over 1.2 million connections, I keep bumping up the limit but the table just continues to grow (but not monotonically -- it does go down sometimes).

$ cat /proc/sys/net/ipv4/netfilter/ip_conntrack_count
1278865

This is true despite the fact that netstat is quite reasonable:

$ netstat -ant | wc -l
908

I am running Debian with Kernel 2.6.32-5-amd64. I understand that the conntrack table keeps recent connection information and so it is expected that it would be larger than netstat, but this behavior still seems rather extreme! The box is being used primarily as a web server, and mod_python in Apache is being used to handle connections. There should be no connections outside of these. There are no excess mod_python threads running (ps -ef looks normal, top looks normal), and the error logs from Apache and mod_python look normal. (I'm unable to post detailed error log information for privacy reasons).

In ip_conntrack, the IP addresses seem to be fairly distributed and on port 443 (as one would expect with a web server serving exclusively over HTTPS). I'm not sure what might be causing the runaway IP connections and I can't find information on the Internet suggesting this is a known issue. Is there an Apache configuration or anything that might be the culprit? Other ideas? Many thanks.

Solution

To answer your question you can set net.netfilter.nf_conntrack_tcp_timeout_established in sysctl.conf The default is like 5 days, which can be dramatically lowered with out affecting any likely 443 traffic. net.ipv4.netfilter.ip_conntrack_max can also be bumped up.

Comments

[Dom]

Could you check with "netstat -antu | wc -l" (with u) to have UDP too ? It can be DNS connections or something else...

[Jenny D]

I find the question a bit unclear. Do you believe that ip_conntrack is storing addresses that don't connect to you, do you believe that someone is spoofing addresses, or what? What do you mean by runaway connections? One thing you could do is to run the apache logs through e.g. awstat to see how many unique addresses are actually connecting to you; that might explain it.

[Dan]

Thanks. "netstat -antu | wc -l" yields 736 (connections are basically all tcp, only a tiny number of udp connections for mosh). I don't have a prior set of beliefs about what is going on, hence my puzzlement. The issue isn't the number of distinct addresses connecting, but that the connections should NOT be persistent. There is a single POST request that the client occasionally makes and that's it. Is there a recommended way to set a timeout value for connections in conntrack?

[phuqm]

Are you using a load balancer with Direct Routing? (With Direct Source Routing, such that the outgoing connection does not go back through the load balancer, but instead "direct"ly to the client?

[Dan]

Thanks, I've done this and it gives similar information. If you know about a recommended way to check or set a timeout value that would be greatly appreciated.