Web application inside Tomcat 7 stops responding after 12-18 hrs

Question

I have an application hosted using tomcat and it stops responding after 12-18hrs, i can access other application like tomcat manager i.e tomcat process is working but my deployed app stops responding.

I could not find anything really significant in logs that I can directly get some reason.

I am using struts2 + spring + hibernate + jasper + jquery. And my tomcat is 7.0.32 hosted th mysql database.

Please let me know - how to know what exactly is the reason i.e debug it out and later how to fix it.

If required I can add some logs, but not sure which logs to add and which not so requesting moderators not to mark this question as false or unanswerable question but comment out and I will add/update the question as required.

I have read some related question but they are really different than mine.

Thanks in advance.

catilina.out was created after server was stopped by me and shows following contents in brief

INFO: Waiting for 200 instance(s) to be deallocated
        Jul 28, 2013 8:27:15 PM org.apache.catalina.core.StandardWrapper unload
        INFO: Waiting for 200 instance(s) to be deallocated
        Jul 28, 2013 8:27:15 PM org.apache.catalina.core.StandardWrapper unload
        INFO: Waiting for 1 instance(s) to be deallocated
        Jul 28, 2013 8:27:16 PM org.apache.catalina.core.StandardWrapper unload
        INFO: Waiting for 1 instance(s) to be deallocated
        Jul 28, 2013 8:27:17 PM org.apache.catalina.core.StandardWrapper unload
        INFO: Waiting for 1 instance(s) to be deallocated
        Jul 28, 2013 8:27:17 PM org.apache.catalina.core.ApplicationContext log
        INFO: Closing Spring root WebApplicationContext
        Jul 28, 2013 8:27:17 PM org.apache.catalina.loader.WebappClassLoader clearReferencesJdbc
        SEVERE: The web application [] registered the JDBC driver [com.mysql.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
        Jul 28, 2013 8:27:17 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
        SEVERE: The web application [] appears to have started a thread named [MySQL Statement Cancellation Timer] but has failed to stop it. This is very likely to create a memory leak.
        Jul 28, 2013 8:27:17 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
        SEVERE: The web application [] appears to have started a thread named [Timer-0] but has failed to stop it. This is very likely to create a memory leak.
        Jul 28, 2013 8:27:17 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
        SEVERE: The web application [] appears to have started a thread named [com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0] but has failed to stop it. This is very likely to create a memory leak.
.............



SEVERE: The web application [] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@55cd6225]) and a value of type [org.springframework.aop.framework.Cglib2AopProxy.CglibMethodInvocation] (value [ReflectiveMethodInvocation: public java.util.List com.cursive.eclinic.processor.DoctorProcessor.list(com.cursive.eclinic.DoctorCriteria) throws java.lang.Exception; target is of class [com.cursive.eclinic.processor.DoctorProcessor]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
        Jul 28, 2013 8:27:18 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
        SEVERE: The web application [] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@1ee69725]) and a value of type [com.opensymphony.xwork2.ActionContext] (value [com.opensymphony.xwork2.ActionContext@56a2f846]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
        Jul 28, 2013 8:27:18 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
        SEVERE: The web application [] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@2513679]) and a value of type [org.apache.struts2.dispatcher.Dispatcher] (value [org.apache.struts2.dispatcher.Dispatcher@21cf431f]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
        Jul 28, 2013 8:27:18 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
        SEVERE: The web application [] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@55cd6225]) and a value of type [org.springframework.aop.framework.Cglib2AopProxy.CglibMethodInvocation] (value [ReflectiveMethodInvocation: public java.util.List com.cursive.eclinic.processor.TimeSlotProcessor.list(com.cursive.eclinic.TimeSlotCriteria) throws java.lang.Exception; target is of class [com.cursive.eclinic.processor.TimeSlotProcessor]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
        Jul 28, 2013 8:27:18 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
        SEVERE: The web application [] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@1ee69725]) and a value of type [com.opensymphony.xwork2.ActionContext] (value [com.opensymphony.xwork2.ActionContext@6559e819]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
        Jul 28, 2013 8:27:18 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
        SEVERE: The web application [] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@2513679]) and a value of type [org.apache.struts2.dispatcher.Dispatcher] (value [org.apache.struts2.dispatcher.Dispatcher@21cf431f]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
        Jul 28, 2013 8:27:18 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
        SEVERE: The web application [] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@55cd6225]) and a value of type [org.springframework.aop.framework.Cglib2AopProxy.CglibMethodInvocation] (value [ReflectiveMethodInvocation: public java.util.List com.cursive.eclinic.processor.HospitalDoctorProcessor.list(com.cursive.eclinic.HospitalDoctorCriteria) throws java.lang.Exception; target is of class [com.cursive.eclinic.processor.HospitalDoctorProcessor]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
        Jul 28, 2013 8:27:18 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
        SEVERE: The web application [] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@1ee69725]) and a value of type [com.opensymphony.xwork2.ActionContext] (value [com.opensymphony.xwork2.ActionContext@7194f54a]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
        Jul 28, 2013 8:27:18 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
        SEVERE: The web application [] created a ThreadLocal with key of ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@2513679]) and a value of type [org.apache.struts2.dispatcher.Dispatcher] (value [org.apache.struts2.dispatcher.Dispatcher@21cf431f]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
        Jul 28, 2013 8:27:18 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
        SEVERE: The web application [] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@55cd6225]) and a value of type [org.springframework.aop.framework.Cglib2AopProxy.CglibMethodInvocation] (value [ReflectiveMethodInvocation: public java.util.List com.cursive.eclinic.processor.HospitalProcessor.list(com.cursive.eclinic.HospitalCriteria) throws java.lang.Exception; target is of class [com.cursive.eclinic.processor.HospitalProcessor]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
        Jul 28, 2013 8:27:18 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks



INFO: Illegal access: this web application instance has been stopped already.  Could not load org.apache.log4j.spi.ThrowableInformation.  The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
        java.lang.IllegalStateException
            at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1599)
            at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1558)
            at org.apache.log4j.spi.LoggingEvent.<init>(LoggingEvent.java:154)
            at org.apache.log4j.Category.forcedLog(Category.java:388)
            at org.apache.log4j.Category.log(Category.java:853)
            at com.mchange.v2.log.log4j.Log4jMLog$Log4jMLogger.log(Log4jMLog.java:161)
            at com.mchange.v2.log.log4j.Log4jMLog$Log4jMLogger.log(Log4jMLog.java:173)
            at com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:629)
            at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:477)
            at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:525)
            at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:128)
            at org.hibernate.connection.C3P0ConnectionProvider.getConnection(C3P0ConnectionProvider.java:78)
            at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:446)
            at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:167)
            at org.hibernate.jdbc.AbstractBatcher.prepareQueryStatement(AbstractBatcher.java:161)
            at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1577)
            at org.hibernate.loader.Loader.doQuery(Loader.java:696)
            at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:259)
            at org.hibernate.loader.Loader.doList(Loader.java:2232)
            at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2129)
            at org.hibernate.loader.Loader.list(Loader.java:2124)
            at org.hibernate.loader.criteria.CriteriaLoader.list(CriteriaLoader.java:118)
            at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1599)
            at org.hibernate.impl.CriteriaImpl.list(CriteriaImpl.java:306)

Edited --------

Tomcat manager server status shows under

"http-bio-80" Max threads: 200 
Current thread count: 149 
Current thread busy: 146 
Max processing time: 1066526 ms 
Processing time: 26227.367 s 
Request count: 5364 
Error count: 262 
Bytes received: 0.00 MB 
Bytes sent: 127.51 MB

And this is memory status

JVM

Free memory: 36.16 MB Total memory: 362.78 MB Max memory: 494.93 MB
Memory Pool Type    Initial Total   Maximum Used
Eden Space  Heap memory 68.31 MB    100.12 MB   136.50 MB   81.74 MB (59%)
Survivor Space  Heap memory 8.50 MB 12.50 MB    17.06 MB    12.50 MB (73%)
Tenured Gen Heap memory 170.68 MB   250.16 MB   341.37 MB   232.37 MB (68%)
Code Cache  Non-heap memory 2.43 MB 11.12 MB    48.00 MB    10.99 MB (22%)
Perm Gen    Non-heap memory 20.75 MB    69.87 MB    82.00 MB    69.76 MB (85%)

What does it mean?

I did a smart thing today restarted tomcat before it stopped responding and got following logs do they say anything

Jul 30, 2013 8:37:32 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["http-bio-80"]
Jul 30, 2013 8:37:32 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["http-bio-443"]
Jul 30, 2013 8:37:32 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["ajp-bio-8009"]
Jul 30, 2013 8:37:32 PM org.apache.catalina.core.StandardService stopInternal
INFO: Stopping service Catalina
Jul 30, 2013 8:37:33 PM org.apache.catalina.core.ApplicationContext log
INFO: Closing Spring root WebApplicationContext
Jul 30, 2013 8:37:33 PM org.apache.catalina.loader.WebappClassLoader clearReferencesJdbc
SEVERE: The web application [] registered the JDBC driver [com.mysql.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
Jul 30, 2013 8:37:33 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [] appears to have started a thread named [MySQL Statement Cancellation Timer] but has failed to stop it. This is very likely to create a memory leak.
Jul 30, 2013 8:37:33 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [] appears to have started a thread named [Timer-0] but has failed to stop it. This is very likely to create a memory leak.
Jul 30, 2013 8:37:33 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [] appears to have started a thread named [com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0] but has failed to stop it. This is very likely to create a memory leak.
Jul 30, 2013 8:37:33 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [] appears to have started a thread named [com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1] but has failed to stop it. This is very likely to create a memory leak.
Jul 30, 2013 8:37:33 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [] appears to have started a thread named [com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2] but has failed to stop it. This is very likely to create a memory leak.
Jul 30, 2013 8:37:33 PM org.apache.catalina.core.ApplicationContext log
INFO: Closing Spring root WebApplicationContext
Jul 30, 2013 8:37:33 PM org.apache.catalina.loader.WebappClassLoader clearReferencesJdbc
SEVERE: The web application [/eclinic] registered the JDBC driver [com.mysql.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
Jul 30, 2013 8:37:33 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/eclinic] appears to have started a thread named [MySQL Statement Cancellation Timer] but has failed to stop it. This is very likely to create a memory leak.
Jul 30, 2013 8:37:33 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["http-bio-80"]
Jul 30, 2013 8:37:33 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["http-bio-443"]
Jul 30, 2013 8:37:33 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["ajp-bio-8009"]

Solution 1

This problem is probably a connection issue and not the problem of tomcat. While using hibernate you should use connection pooling also.Otherwise there is chance to disconnect from database and connection couldnot establish after a certain time period. The utility jars like C3P0 are available for solving issue. Also please check "catalina.out" file inside tomcat folder for the log.

Solution 2

Though I am not sure about the order of your logs it seems that your application is failing in the following order:

• Some hiberante queries fail to execute and throw an exception
• The exceptions are not handled appropriately and possibly leave critical resources open (i.e. you have memory leaks - perhaps it does not close the database connections or the hibernate sessions).
• This causes some threads to hang (most of them seem to belong to third party libraries but some are suspects that are created by your application).
• Tomcat detects those conditions and is trying to shut down the corresponding services trying to reclaim the server's stability and eventually shuts down your applciation.

If you have the source code check if all database related code works correctly (i.e. connections/session are closed in a finally block no matter what) and check that threads created by the application (if any) will shutdown gracefully.