Stale Network connections to Tomcat AJP port

Question

I have Tomcat Application server connected with Apache server using mod_jk connector. Both are running on different servers.

Apache ----->------ Firewall ----->------ Tomcat

For the last couple of days, I am seeing the thread count in the Tomcat AJP connector pool is getting filled (100%) And I am seeing error in Apache server.

Troubleshooting Steps : I have stopped Apache server and ran following command on Apache server

> netstat -an | grep 8009 | wc -l
0

Then, I ran the same command on Tomcat server and I could see lots of connections from Apache to AJP port is still in established state.

> netstat -an | grep 8009
tcp        0      0 :::8009                     :::*                        LISTEN      
tcp        0      0 ::ffff:192.168.1.75:8009     ::ffff:192.168.10.75:56840   ESTABLISHED 
tcp        0      0 ::ffff:192.168.1.75:8009     ::ffff:192.168.10.75:56838   ESTABLISHED 
---deleted remaining lines----

I have waited for 1-2 hours and I could see these stale connections are still there.

I have following Tomcat Settings :

tomcat.maxthreads=200
tomcat.minsparethreads=50
tomcat.maxidletime=10000
tomcat.acceptcount=100

Thread dump showing following Threads :

"ajp-bio-8009-exec-70" daemon prio=10 tid=0x00007fb87c3a1800 nid=0x302b runnable [0x00007fb8605c4000]
   java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:152)
    at java.net.SocketInputStream.read(SocketInputStream.java:122)
    at org.apache.coyote.ajp.AjpProcessor.read(AjpProcessor.java:312)
at org.apache.coyote.ajp.AjpProcessor.readMessage(AjpProcessor.java:367)
at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:118)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314)
- locked <0x000000051b0a2ee0> (a org.apache.tomcat.util.net.SocketWrapper)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)

I need to restart to clean all the stale connections to Tomcat.

Please help me in troubleshooting this issue. Is this due to Server issue ?or Tomcat or mod_jk issue.?

Solution

By default, mod_jk keeps all ajp13 connections open indefinitely but it does not send keepalives across the tcp session to the tomcat server. If that connection is idle, it will remain open. Firewalls however do not like idle sessions and after a period of inactivity, will sever that connection. This is why the initial connection to the application might hang. ajp13 will hand the connection off to a tcp connection that is currently open but the firewall has killed that connection.

Try add in workers.properties, this parameters for each worker:

worker.ajp13.socket_keepalive=True
worker.ajp13.connection_pool_timeout=300

In server.xml on the tomcat ajp13 connector section, add the connectionTimeout parameter:

<!-- Define an AJP 1.3 Connector on port 8009 -->
<Connector port="8009" protocol="AJP/1.3" redirectPort="8443"
connectionTimeout="300000" />  

I hope this help.

Comments

[ArunS]

Thanks. connectionTimeout="300000" solved my issue. The stale connections are getting timedout after "connectionTimeout" value.