The web application appears to have started a thread named [Timer-0] but has failed to stop it

Question

I am using Spring Boot 1.5.9.RELEASE + Java 8 + Tomcat 9 + Jersey + Oracle and my app has scheduled method defined as follows:

@Configuration
@EnableScheduling
public class ScheduleConfig implements SchedulingConfigurer {

    @Override
    public void configureTasks(ScheduledTaskRegistrar taskRegistrar) {
        taskRegistrar.setScheduler(taskExecutor());
    }

    @Bean(destroyMethod = "shutdown")
    public Executor taskExecutor() {
        return Executors.newScheduledThreadPool(100);
    }
}

The job class:

@Component
public class ClearCacheJob {



    @Scheduled(fixedRate = 3600000, initialDelay = 10000)
    public void clearErrorCodesCache() {
        try {
            logger.info("######## ClearCacheJob #########");
        } catch (Exception e) {
            logger.error("Exception in ClearCacheJob", e);
        }
    }

}

Also I have a class to deregister the Oracle driver as follows:

@WebListener
public class ContainerContextClosedHandler implements ServletContextListener {

    private static final Logger logger = LoggerFactory.getLogger(ContainerContextClosedHandler.class);

    @Override
    public void contextInitialized(ServletContextEvent servletContextEvent) {
        logger.info("######### contextInitialized #########");
    }

    @Override
    public void contextDestroyed(ServletContextEvent servletContextEvent) {
        logger.info("######### contextDestroyed #########");
        Enumeration<Driver> drivers = DriverManager.getDrivers();
        while (drivers.hasMoreElements()) {
            Driver driver = drivers.nextElement();
            try {
                DriverManager.deregisterDriver(driver);
                logger.info(String.format("deregistering jdbc driver: %s", driver));
            } catch (SQLException e) {
                logger.info(String.format("Error deregistering driver %s", driver), e);
            }

        }
    }

}

but when stopping Tomcat I am getting the following error:

WARNING [Thread-11] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [hai] 
appears to have started a thread named [Timer-0] but has failed to stop it. 
 This is very likely to create a memory leak. Stack trace of thread:
 java.lang.Object.wait(Native Method)
 java.lang.Object.wait(Unknown Source)
 java.util.TimerThread.mainLoop(Unknown Source)
 java.util.TimerThread.run(Unknown Source)

Why am I getting this error and how can I fix it?

Solution 1

I want to share some solutions with root cause analysis of this issue.

---

For Oracle Users

Solution #1

You should remove your Oracle driver from Tomcat's /lib folder. I was facing the same issue and it got resolved.

Note: Let the Oracle driver be in /WEB-INF/lib folder.

Solution #2

You can use real hack by sleeping thread.

@Override
public void contextDestroyed(ServletContextEvent servletContextEvent) {
    logger.info("######### contextDestroyed #########");
    Enumeration<Driver> drivers = DriverManager.getDrivers();
    while (drivers.hasMoreElements()) {
        Driver driver = drivers.nextElement();
        try {
            DriverManager.deregisterDriver(driver);
            logger.info(String.format("deregistering jdbc driver: %s", driver));
        } catch (SQLException e) {
            logger.info(String.format("Error deregistering driver %s", driver), e);
        }
    }
    try { Thread.sleep(2000L); } catch (Exception e) {} // Use this thread sleep
}

Resource Link: Solution to “Tomcat can’t stop [Abandoned connection cleanup thread]”

Solution #3

Svetlin Zarev has told nothing to worry about. It is the standard message of tomcat. He has given root cause analysis like below:

This problem is occurred when an application has started ScheduledExecutor (but this will happen with any other Thread/TheadPool) and didn't shut it down on contextDestroyed. So check if you are shutting down your threads on application/server stop.

Resource Link: Tomcat8 memory leak

Solution #4

For Oracle users, there are multiple answers in this post: To prevent a memory leak, the JDBC Driver has been forcibly unregistered

---

For MySQL users

Solution #5

Root Cause Analysis with Solution:

The cleanup thread for abandoned connections in the NonRegisteringDriver class was refactored to have a static shutdown method. Memory was allocated but never released. If you encountered this leak problem, implement the context listener in your application with the AbandonedConnectionCleanupThread.shutdown() call in the contextDestroyed method.

This issue was found in applications running under the Tomcat application server, but it might have also applied to other application servers.

For example:

@WebListener
public class YourThreadsListener implements ServletContextListener {
   public void contextDestroyed(ServletContextEvent arg0) {
      try {
          AbandonedConnectionCleanupThread.shutdown();
      } catch (InterruptedException e) {
      }
   }
   ...
}

Note that if container does not support annotations, you add the description to web.xml:

<listener>
    <listener-class>user.package.YourThreadsListener</listener-class> 
</listener>

Resource Link: https://docs.oracle.com/cd/E17952_01/connector-j-relnotes-en/news-5-1-23.html

Solution 2

Change your ScheduleConfig to use shutdownNow instead of shutdown as destroy method.

@Configuration
@EnableScheduling
public class ScheduleConfig implements SchedulingConfigurer {

    @Override
    public void configureTasks(ScheduledTaskRegistrar taskRegistrar) {
        taskRegistrar.setScheduler(taskExecutor());
    }

    @Bean(destroyMethod = "shutdownNow")
    public Executor taskExecutor() {
        return Executors.newScheduledThreadPool(100);
    }
}

Solution 3

My conclusions after running a few tests based on your code and researching online:

• There's nothing to worry about (link). Tomcat process is being finished and there's no memory leaks left behind.

• Even if you call something like AbandonedConnectionCleanupThread.shutdown(), you could still get that same Warning (link)

• This warning happens when calling startup.sh and shutdown.sh. When running Tomcat from Eclipse, it doesn't show that Warning.

• Your shutdown method for the Executor is likely being called. For my tests, it was getting called even if I didn't define the destroyMethod for the executor.

• In this case, this warning is not related to any Spring Scheduling bean. Executors.newScheduledThreadPool returns a new ScheduledThreadPoolExecutor, which has the destroy method and it is getting destroyed, like I pointed out earlier. You can debug and see it for yourself.

• However, there's somewhere at your code calling new java.util.Timer, which calls new TimerThread(), ass seen from your logging, and as pointed out by @Claudio Corsi.

In order to debug it and if you are using Eclipse, you have to attache the source code for your JDK version. Open the class declaration (hold ctrl and choose open declaration) and click the "Attach Source Code" button. Make sure you have dowloaded the exact same version. You don't even have to extract the zip. If you're using Maven, just hold on a bit that it will download for itself.

Then, place a breakpoint in the constructor for java.util.Timer and start debugging your application.

Edit: After identifying a reference to java.util.Timer, save it (as a bean, if it's not one) and call its cancel method on context destroy.

Solution 4

It is hard to say root cause but thread name [Timer-0] gives a clue to find it. java.util.Timer class creates threads which has name pattern like Timer-* as you can see in it's source code.

public Timer() {
    this("Timer-" + serialNumber());
}

Possibly the libraries that are in your classpath starts a Timer thread but doesn't cancel it or the code which is working in this thread stuck.

I may suggest put breakpoint in java.util.Timer and debug it to find which tasks is working on it. It may point the root cause.

Solution 5

I have also got the same issue with following error:

The web application [ROOT] appears to have started a thread named [cluster-ClusterId{value='5d29b78967e4ce07d9bb8705', description='null'}-localhost:27017] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:

So, after a while I figured it out that I didn't do the maven install for all sub modules in my spring-boot application. So, double check if you are having same error that:

1. You have run mvn clean install -U for all sub modules in the project and for the project itself too.

View more solutions

Comments

[Mahmoud Saleh]

about the AbandonedConnectionCleanupThread, is it for mysql only ?

[SkyWalker]

@MahmoudSaleh, yes it is.

[Jörg]

you can also try a breakpoint in your contextDestroyed() and use e.g. jvisualvm to show thread dump of [Timer-0], maybe you get a clue from its stack about what part is responsible for starting it and fails to also have it stop