Apache + Tomcat with mod_jk - Web site hangs

Question

I have a website with apache 1.3(SSL enabled) + mod_jk + tomcat 5.5 on Linux redhad setup. Just recently i started having a downtime problem with my web site. Once a day, i get my web site hang on port 80. But if i access directly through 8080 tomcat responses and web site works fine. Both 80 and 8080 ports are accessible but apache and tomcat connection with mod_jk is broken. Only after i restart tomcat, the things get back to normal.

I just configured apache mod_Jk to log the errors, so i will see if there will be any error on the next hang.

apache mod_js conf:

JkShmSize   1000M

apache workers conf:

worker.list=worker1
worker.worker1.type=ajp13
worker.worker1.host=127.0.0.1
worker.worker1.port=8009
worker.worker1.lbfactor=1
worker.worker1.socket_keepalive=1
worker.worker1.recycle_timeout=180
worker.worker1.sticky_session=False

I checked web application error/warn logs on tomcat, i have some "out of memory" java exceptions. Can application errors cause this issue ? Can it be website overload problem or memory leak ? Currently dev/mapper/VolGroup00-LogVol00 has only 4% free space. Can it be a cause for the problem ?

I also got this log entry, it matchs the server hang time:

/var/log/messages: possible SYN flooding on port 8009. Sending cookies

Update: I just got another down, the mod_jk log gives this:

[Sun Nov 14 00:57:03 2010] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (961): Can't receive the response message from tomcat, network problems or tomcat is$ [Sun Nov 14 00:57:03 2010] [error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet) [Sun Nov 14 00:57:08 2010] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (961): Can't receive the response message from tomcat, network problems or tomcat is$ [Sun Nov 14 00:57:08 2010] [error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet) [Sun Nov 14 00:57:12 2010] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (961): Can't receive the response message from tomcat, network problems or tomcat is$ [Sun Nov 14 00:57:12 2010] [error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet) [Sun Nov 14 00:57:12 2010] [error] ajp_service::jk_ajp_common.c (1758): Error connecting to tomcat. Tomcat is probably not started or is listening on the wrong port. w$ [Sun Nov 14 00:57:12 2010] worker1 mydomain.com 50.999342

Looks my website is down on 8009 port. AJP 1.3 connector serves apache mod_jk on 8009 port.

Any help or advice would be highly appreciated.

Thanks.

Solution

You need more data.

1) setup the server-status handler for apache

2) setup the status worker of mod_jk. With this you'll then be able to clearly see whether this is a apache to tomcat communication problem.

3) when tomcat hangs again do a thread dump to see where the Java processing is currently at. Maybe it's waiting for something.

Edit: In case it's a problem of apache to tomcat problem ([2]) have a look at http://community.jboss.org/wiki/OptimalModjk12Configuration