Tomcat doesn't stop. How can I debug this?

Question

I have a Tomcat 7 running in Linux that I start via $CATALINA_HOME/bin/startup.sh and shutdown via $CATALINA_HOME/bin/shutdown.sh
from /etc/init.d

All is ok except 1 problem. Sometimes tomcat does not stop.
Although I stop it and I see in catalina.out logs that is going down, if I do ps -ef I can still see the process running.

What could be the problem? How can I debug this? My feeling is, that this is related to threads.

So the parts that are suspicious are the following:
1) I use Log4j's LogManager to detect if the log4j configuration has been changed, but I do Log4jManager.shutdown on a contextDestroyed ServletContextListener
2) I use H2 database and I see on shutdown:

SEVERE: The web application [/MyApplication] appears to have started a
thread named [H2 Log Writer MYAPPLICATION] but has failed to stop it.
This is very likely to create a memory leak

SEVERE: The web application [/MyApplication] appears to have started a
thread named [H2 File Lock Watchdog
/opt/myOrg/tomcat/webapps/MyApplication/db/myDatabase.lock.db] but has
failed to stop it. This is very likely to create a memory leak. Apr 2,
2012 9:08:08 AM org.apache.catalina.loader.WebappClassLoader
clearReferencesThreads SEVERE: The web application [/MyApplication]
appears to have started a thread named [FileWatchdog] but has failed
to stop it. This is very likely to create a memory leak.

Any help please? How can I detect the problem here?

UPDATE:
I did a kill -3 as suggested by @daveb, and in the catalina.out I see:

JVMDUMP006I Processing dump event "user", detail "" - please wait. JVMDUMP032I JVM requested Java dump using '/etc/init.d/javacore.20120402.093922.2568.0001.txt' in response to an event JVMDUMP010I Java dump written to /etc/init.d/javacore.20120402.093922.2568.0001.txt JVMDUMP013I Processed dump event "user", detail "".

There is a javacore in /etc/init.d but I don't know how to process it. I.e. what parts should I investigate

Solution 1

If the web application is stopped, all connections to the database should be closed as well. If you don't have a list of connections, then execute the SQL statement "shutdown" (this only works for the H2 and HSQLDB databases).

If you have a registered a Servlet, you can do that in the Servlet.destroy() method.

If you have registered a ServletContextListener, you can execute the "shutdown" statement in the ServletContextListener.contextDestroyed(ServletContextEvent servletContextEvent) method. This is what org.h2.server.web.DbStarter ServletContextListener does (the one that is included in the H2 database).

Solution 2

Find out what threads are still running (or blocked, waiting to run) by using jstack or sending a signal to the process:

kill -3 pid

When you know this, you can make whatever it was that started them hook into the shutdown notification to stop the threads. Or make those threads deamon threads.

See This tomcat shutdown question for more details on this.

If you don't know where your threads were created, then consider adding names to them - executors can take thread factories, and you can use those factories to set the deamon status of a thread and also to name it - so your stack trace will be clearer.

Solution 3

Check if your Web Application has some Scheduler active, like Quartz.

If you don't stop it, Web Application Thread never ending until you kill it

Solution 4

I had the exact same problem. Sometimes, the command ./shutdown.sh does not stop the tomcat process, and its java process stays in the running processes.

I had solved this problem using the Tomcat version in the Ubuntu's software repositories, by:

sudo apt-get install tomcat7

After installing it from package manager and configuring some settings, I did not have any problems on stopping/starting Tomcat. I used this command to stop, and it never failed:

service tomcat7 stop

which is nearly the same as

/etc/init.d/tomcat7 stop

Using this command runs the code block from the init script, specifically, the codes from the file /etc/init.d/tomcat7. So I looked into it to see what it does to always kill the tomcat process succesfully. Here is the code block that runs when you use service tomcat7 stop command:

log_daemon_msg "Stopping $DESC" "$NAME"

        set +e
        if [ -f "$CATALINA_PID" ]; then
                start-stop-daemon --stop --pidfile "$CATALINA_PID" \
                        --user "$TOMCAT7_USER" \
                        --retry=TERM/20/KILL/5 >/dev/null
                if [ $? -eq 1 ]; then
                        log_progress_msg "$DESC is not running but pid file exists, cleaning up"
                elif [ $? -eq 3 ]; then
                        PID="`cat $CATALINA_PID`"
                        log_failure_msg "Failed to stop $NAME (pid $PID)"
                        exit 1
                fi
                rm -f "$CATALINA_PID"
                rm -rf "$JVM_TMP"
        else
                log_progress_msg "(not running)"
        fi
        log_end_msg 0
        set -e
        ;;

The important part is this:

start-stop-daemon --stop --pidfile "$CATALINA_PID" \
                            --user "$TOMCAT7_USER" \
                            --retry=TERM/20/KILL/5 >/dev/null

This means "retry stopping until the process is stopped. Here is the --retry command documentation from start-stop-daemon manual:

   -R|--retry timeout|schedule
          With  --stop,  specifies  that  start-stop-daemon  is  to  check
          whether  the  process(es)  do  finish.  It will check repeatedly
          whether any matching processes are running, until none are.   If
          the  processes  do  not exit it will then take further action as
          determined by the schedule.

          If timeout is specified instead of schedule  then  the  schedule
          signal/timeout/KILL/timeout  is used, where signal is the signal
          specified with --signal.
          ...

So, --retry=TERM/20/KILL/5 means "Send TERM signal to the process, wait 20 seconds, if it's still running, send KILL signal, wait 5 seconds, if it's still running, there is a problem.

This means you can configure the tomcat to run as a deamon and use a command like this, or write a script to do that kind of action to stop tomcat, or just use Ubuntu and get the tomcat from the package manager.

Solution 5

I also had the same problem. There was a ThrottledThreadPoolExecutor in my application that wasn't getting shutdown. When I shut it down properly, tomcat would stop cleanly. In order to figure out the problem, I had to remove all the apps from my tomcat webapps directory and then add them one by one and see which one was causing the problem

View more solutions

Comments

[daveb]

Open the javacore .txt file in a text editor, look for thread stacks that include packages that you've written.

[Jim]

Yes I figured that.But I am not sure on what indicates an error.

[Jim]

Thanks for the reply!If the web application is stopped, all connections to the database should be closed as well. I am using Tomcat's connection pool. So it is out of my hands.Is this a known 'racing' issue between Tomcat and H2.I can do SHUTDOWN (it is safe right?) but I want to make sure I address this correctly

[Thomas Mueller]

It see the problem. Not sure what the best solution is... ignore the exception? The statement SHUTDOWN will only close this database, so it should be relatively save - only if you are not sure if other web apps are using the database, then you can't do that. Another solution is to use the server mode (run the H2 database in another process).

[Jim]

:No the database is mine i.e. no other applications will access it.And I am required to use it in file mode.So basically you are saying that 1) shutdown is safe for the data 2)I can ignore the exception meaning what?That Tomcat does not hang on shutdown because of H2 still running?I see the db.lock file in the directory of my app

[Thomas Mueller]

The <databaseName>.lock.db file is not deleted as long as the database is open. The database is open because there is at least one open connection. There is at least one open connection because the connection pool is not disposed. If you execute shutdown then the database is closed. In your case, that is save.

[Vicky]

In my case i have to stop all the connection including quartz thread when tomact stops.Even i have ServletContextListener.contextDestroyed method but i dont know how to stop all connections when tomcat stops

[hello_earth]

there's also this useful thread: serverfault.com/questions/1021145/…. and oh my was I so sure that I was spawning no threads of mine - and yet I really did, I just forgot. I was so sure that I was searching for old bug reports on legacy iBatis that we are using, thinking that it was the culprit - not cleaning up connections and such. was I so wrong.