How to mitigate error "kernel: nf_conntrack: table full, dropping packet"

Question

We recently had a problem with one of our servers (Debian Squeeze) becoming unresponsive during heavy-ish load. Looking at the kernel logs, I think this is the cause:

kernel: nf_conntrack: table full, dropping packet

As I understand it, this is the conntrack module, which does some stateful tracking of connection, reporting that the table used to store the connection details is full.

From the research I have done, there seem to be two ways to mitigate this:

1. Increase size of the table.

2. Remove the module from the system altogether.

However, neither /proc/sys/net/ipv4/ip_conntrack_maxnor /proc/sys/net/ipv4/netfilter/ip_conntrack_max exist on this machine (there is no ipv4 catalogue under net).

If I do lsmod I get no results.

So, I'm a bit confused - perhaps someone could clarify the situation for me?

• Is conntrack installed? If so, where are the settings? And why doesn't it show up in lsmod?
• If conntrack is not installed, what is issuing the table full messages?

Thank you

Solution 1

This is from experience - I haven't done research to verify this information: I have seen a few systems where this same error is in the system logs and there is nothing in /proc/sys/net/ipv4/ip_conn* or /proc/sys/net/ipv4/netfilter. I would also like to know why - but that dosn't remain very important once you find a fix for the original symptoms. ;)

The mitigation strategy was twofold: Increasing the limit via sysctl (naive short-term approach) and to figure out why the number of connections being tracked is so high.

If the default limits are being exceeded and the server in question isn't intended to handle high number of connections then it stands to reason that the limits shouldn't be hit at all. A good example of a service that will have high connection tracking requirements is a "public" DNS server servicing a hundred-thousand-plus clients.

The mitigation would be to look into logs, ensure anti-DOS/DDOS measures are in place (for example see fail2ban), and ensure that you have a sensible firewall configuration installed.

Regarding lsmod, I haven't come across the situation where it appears to be active but the module isn't listed. I'm not certain of how that situation arises.

Solution 2

The settings for conntrack are often in /proc/sys/net/netfilter/nf_conntrack_max.