Is there a way to avoid undeployment memory leaks in Tomcat?

Question

This question is for anyone who's ever tested the "Find leaks" button in the Tomcat manager and got some results like this:

The following web applications were stopped (reloaded, undeployed), but their classes from previous runs are still loaded in memory, thus causing a memory leak (use a profiler to confirm):
/leaky-app-name

I'm assuming this has something to do with that "Perm Gen space" error you often get with frequent redeployments.

So what I'm seeing in jconsole when I deploy is that my loaded classes goes from about 2k to 5k. Then you would think an undeployment should drop them back down to 2k but they remain at 5k.

I've also tried using the following JVM options:

-XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC -XX:+CMSPermGenSweepingEnabled

I did see VERY minor dips in the amount of Perm Gen space used but not what I expected and the loaded class counts did not drop.

So is there a way to configure Tomcat or design your app to unload better on an undeployment? Or are we stuck with restarting the server after some major debugging sessions?

Tomcat version output:

Server version: Apache Tomcat/6.0.29
Server built: July 19 2010 1458
Server number: 6.0.0.29
OS Name: Windows 7
OS Version: 6.1
Architecture: x86
JVM Version: 1.6.0_18-b07
JVM Vendor: Sun Microsystems Inc.

Update:

Thanks to celias' answer I decided to do a little more digging and I think I determined the culprit to be in my application thanks to CXF, Spring and JAXB.

After I learned how to profile a Java application, I pointed the profiler at Tomcat and took some heap dumps and snapshots to see what the objects and classes looked like in memory. I discovered that some of the enumerations from my XML schema used in my CXF/JAXB (wsdl2java) generated classes were lingering after an undeployment. According to my heap dump it looks like the objects were tied to a Map. Disclaimer: I admit I'm still a little green with profiling and tracing an object's call tree can be challenging in Java.

Also I should mention that I didn't even invoke the service, just deployed then undeployed it. The objects themselves appeared to be loaded via reflection initiated from Spring on deployment. I believe I followed the convention for setting up a CXF service in Spring. So I'm not 100% sure if this is Spring/CXF, JAXB, or reflection's fault.

As a side note: the application in question is a web service using Spring/CXF and the XML happens to be a rather complex schema (an extension of NIEM).

Solution 1

If you want to make sure not to cause leaks you have to do the following:

• Make sure your web application does not use any java classes that are in the web container shared libraries. If you have any shared libraries, make sure there is no strong references to the objects in those libraries
• Avoid using static variables, especially on java objects like HashTable, Sets, etc. If you need to, make sure that you call remove to release the objects with the maps, lists...

Here is also a good article on ThreadLocal and MemoryLeaks - http://blog.arendsen.net/index.php/2005/02/22/threadlocals-and-memory-leaks-revisited/

Solution 2

Tomcat 7 is supposed to bring improvements in this area. See Features of Apache Tomcat 7, section titled No More Leaks!

They believe they can now cope with a lot of memory leaks caused by the web applications. Unfortunately, it's still in beta.

Other than that, I can just say that I've made the same experience and haven't found a solution. Deploying usually requires restarting Tomcat afterwards. I have no idea who the culprit is: my web application, Tomcat, Hibernate, Tapestry or several of them.

Comments

[waltwood]

Good info! I tried making a very simple app without any 3rd party or shared libs and did not see the same symptoms I outlined above. The only problem is that a "real world" app will likely have some memory leaks somewhere with the use of 3rd party libs. So I guess it all comes down to the developers somewhere along the line. I'm sure we could all make improvements: Tomcat, our Apps, Libraries...

[Codo]

Does anybody know more about the effect of dynamic byte code generation as it is used by many XML serializer, Hibernate, Tapestry? Do these libraries properly unload?

[waltwood]

Ok, I was hasty in saying I didn't see the same symptoms. Maybe my test app was too small to notice. I tested with the Tomcat Examples App, and again I saw PermGen filling up and the loaded classes never returned to the baseline after undeployment. I understand that PermGen is used to store metadata about loaded classes. So I would think that an undeployment should cause Tomcat to drop class references for an app and the GC should clear them and their metadata up right? Even if my classes contain strong references to other classes, shouldn't they die when Tomcat unloads the parent classes?

[waltwood]

Sorry for the back-and-forth, but after some more research and a crash course in profiling, I realized I was confusing class unloading with object destruction. I expected to see class counts reducing as I updeployed applications from Tomcat. Please see my update on the question. Also when I hit "find leaks" it did not report anything after undeploying the the example app.