Killing CLOSE_WAIT sockets without killing parent process on Linux

Question

Tomcat is leaving me with CLOSE_WAIT sockets which ultimately saturate the maximum number of connections.

I've tried many methods in my client and server code to get rid of these to no avail, including closing connections, calling System.gc(), etc.

Now I'm trying to find a way to simply time these out quickly in the OS. I've got conntrack working, but am not sure how to use that to kill these connections. I've also set /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_close_wait to 1, which of course is too low but the connections persist.

Is there a way to kill these zombie sockets?

Running Ubuntu.

Solution

I believe CLOSE_WAIT on the server side of the connection means that the server has received a FIN from the client, will have acknowledged this back to the client and then informed the application that it can close the connection.

It is then up to the application to relinquish the connection once it is satisfied that all the data has been read from the connection.

Once it relinquishes the connection the server will send a final FIN back to the client and the connection will be fully closed.

Suggests it has nothing to do with "TCP_tuning"

Are you sure your application is closing the sockets?

When i wrote a python server, i learnt this :D

UPDATE
Depending on your Tomcat version, you may have experienced this problem due to a bug introduced in Tomcat 6 with the keepAliveTimeout feature in Coyote's AJP protocol.
The nature of this problem was caused by Tomcat failing to close sockets after the keepAliveTimeout expired. The Tomcat sockets would remain in a CLOSE_WAIT state but the corresponding mod_jk sockets would close as normal.

his bug was fixed in SVN commit r589062 and released in Tomcat 6.0.15

Comments

[Alex Neth]

My application is Tomcat. I've read in some places, namely here: old.nabble.com/… that Tomcat doesn't always close sockets until the connection is garbage collected, but System.gc() does not seem to do this. Other than that, I have no control over the socket as this is Tomcat. My application calls close() on the connection whenever possible. (This also occurs with Jetty, so it may be a more general Java issue.)

[Arenstar]

use a SingleClientConnManager, and create and shut it down for every request. This did the trick. I assume that this is what's needed to make sure the connection is closed. ( says a post i just found )

[Alex Neth]

I don't think there is a way to do this in a servlet.

[Arenstar]

From what i have read, this is a consistent problem found.. I would attempt to ensure your version doesnt have a resurfaced bug, and write your code in a way which cleans up.. The OS level fix would be a hack- forcing connections closed, you should really address the issue directly

[Alex Neth]

Unfortunately as far as I know I have no access to the raw sockets in a servlet, so I don't know of any way to code around this.