netstat and ip_conntrack connection count differ by order of magnitude. Why?

linux ubuntu networking netstat

14,077

Solution 1

Conntrack module remembers recent connections for X seconds before they finally expire. This, in my understanding, is because iptables has several other modules that can utilize this information: for example, if you want to ban some IP address if it makes X new connections during some time frame.

netstat, on the other hand, shows real-time information and is not interested about ancient history.

Have you increased maximum amount of entries in conntrack table? With a recent-ish kernel, what does

sysctl net.ipv4.netfilter.ip_conntrack_max

... or with some older kernel,

sysctl net.ipv4.ip_conntrack_max

return to you? You may raise that value permanently via /etc/sysctl.conf or temporarily (until next reboot) via sysctl -w net.ipv4.ip_conntrack_max

Solution 2

We stumbled across this case when containers (docker) were in use.

Not sure if it helps in your case or not, but it looks like netstat -nat on the host OS will only show connections intended for the host's networking stack whereas conntrack -L will show information for both the host and all its containers.

If you run netstat -nat from inside the container involved in the connection reported by conntrack -L, you should see the connection information listed there.

14,077

Poma

MSU student

Updated on September 18, 2022

Comments

Poma over 1 year
in /proc/net/ip_conntrack I have:
```
established 3076
time_wait 4346
total 7468
```
and in netstat I have:
```
established 1051
time_wait 73
total 1165
```
Why is that? Where are other connections? How to figure out what are they doing?

Update: Some more stats on ip_conntrack
```
assured 5230
unreplied 2133
total 7427
```
- Greg Petersen over 12 years
  
  Is this box acting as a router?
- Poma over 12 years
  
  no, this is a webserver. It also has nginx as reverse-proxy so there are lots of connections on loopback interface
- Greg Petersen over 12 years
  
  What about the netstat -nat | grep -c 127.0.0.1 and grep -c 127.0.0.1 /proc/net/ip_conntrack?
- Poma over 12 years
  
  493 and 3392 respectively
- Kyle Brandt over 12 years
  
  Not that I don't want to know the answer to this as well, but were you looking into this for a particular reason? Or just exploring?
- Poma over 12 years
  
  We're hitting some connection count limit. Users often can't connect to our website and receiving 500 error.
Kyle Brandt over 12 years

of course, if there are already NOTRACK targets in your raw table, that would by why they differ... but I suspect there is something else as far as lifetime or special loopback handling goes.
Kyle Brandt over 12 years

My only other guess is that perhaps the difference is because of UNREPLIED vs ASSURED? faqs.org/docs/iptables/theconntrackentries.html
Poma over 12 years

cat /proc/sys/net/ipv4/netfilter/ip_conntrack_max = 65536
Poma over 12 years

No, there are no connection table full messages. It can be nginx or filehandle limits, I don't know how to figure this out since I'm relatively new to linux. The main problem is described in this question: serverfault.com/questions/312947/…
Poma over 12 years

I've added this stats to my question