Tomcat randomly shuts down with an AbstractProtocol pause after mild usage

Question

After running my webapp for a while (timing varies between hours and days depending on traffic) Tomcat seemingly randomly shuts itself down. There's nothing out of the ordinary in the log before this happens (no exceptions) just normal INFO stuff that my app emits.

Can anyone help on how best to debug this? Is there anything in Tomcat that would trigger the AbstractProtocol pause signal?

Logs:

09-Nov-2011 21:40:19 org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["http-bio-80"]
09-Nov-2011 21:40:20 org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["ajp-bio-8009"]
09-Nov-2011 21:40:21 org.apache.catalina.core.StandardService stopInternal
INFO: Stopping service Catalina

Java version: 1.6.0_25-b06 Tomcat version:6

Solution 1

I've seen this happen before and the logs can be sometimes cryptic. I do notice though that you're running on Port 80 (which is a whole other issue) which leaves me to believe you modified your server.xml. If there are multiple tomcat instances on the server ensure they all have unique shutdown ports. The standard port is 8005 and if multiple instances are listening on the same port it is possible to take them both down.

There could be a random System.exit() in your code or in a leveraged library or you are improperly suppressing the true error message in your code. I've seen it plenty of times that people write this in controllers as a form of error handling:

try {
  // Do something
} catch(Exception e) {
  // No rethrowing or logging, just suppressing
}

This keeps the true error from bubbling up or even proper logging.

I have also seen tomcat servers just stop functioning when they hit memory limits. There will be a Heap Space error somewhere in the logs but for some reason I find it can get buried pretty easily and all that will be left will be the pause messages. It might be worthwhile to look in the localhost.log besides from the catalina.out. There's usually less garbage in there.

Solution 2

Our tomcat was mysteriously shutting down because our custom script started original startup.sh (without nohup) and went on to do some time consuming stuff. Person who invoked custom script used ctrl-Z to put it in background and then closed putty terminal. This sent SIGHUP to all processes attached to the terminal and tomcat got shutdown.

Wish tomcat had printed a message about receiving SIGHUP. We later found this link which confirmed our analysis.

http://solveissue.com/note?id=1767204

Solution 3

If you are working on production box or want a quick/temporary fix then you might want to set your shutdown port in server.xml to "-1", which will disable the shutdown command. This will not let other apps, which might be having the same shutdown port, send a shutdown signal to your tomcat instance. But with this setting you will have to kill you tomcat instead of shutting it down. That's the bad part.

Comments

[aldux]

There were a lot of specific Error handling, but no general Exception one. That exposed the real problem.

[user2779544]

@Aaron i have same issue, can u pls guide to solve it