How to diagnose a Java 8 metaspace leak?

Question

I have a J2EE application with some interesting behavior ... the heap seems to behave well, growing and shrinking with garbage collections as expected over time. There is no appreciable overall long term heap expansion. However, the metaspace just keeps steadily growing at about 20 Mb per hour until we hit MaxMetaspace and encounter an OOME. I have tried both the parallel and G1 garbage collectors (jdk1.8.0_40).

The application is not getting re-deployed during the execution, so it doesn't seem like it would be the typical classloader leak. Does anyone have suggestions as to how to track down the source of this leak?

Solution 1

The main cause for the java.lang.OutOfMemoryError: Metaspace is:

• either too many classes or
• too big classes being loaded to the Metaspace.

If you want to recreate the problem use this code snippet:

public class Metaspace {
static javassist.ClassPool cp = javassist.ClassPool.getDefault();

public static void main(String[] args) throws Exception {
    for (int i = 0; ; i++) { 
        Class c = cp.makeClass("eu.plumbr.demo.Generated" + i).toClass();
    }
  }
}

All those generated class definitions end up consuming Metaspace.

Javaassist in Maven repo.

You can find a lot more about OOME here

Solution 2

Do a heap dump and analyze it with Eclipse MAT. Look at the classes you have loaded. Check if there's something unexpected, especially duplicate classes. It also has a classloader explorer.

Edit: In theory you could also be that you're constantly generating proxies.

Comments

[It Worked Yesterday]

We tried the MAT classloader explorer. Unfortunately the app is busy and complex enough and the leak was slow enough that it was still "needle in a haystack" proposition. This configuration was intended to be an interim solution anyway and we reached the point of diminishing returns, so we gave up.

[Philippe Marschall]

I'm not aware of a WildFly issue. OP never did debug the issue, so it could be anything including his/her application.

[Grayden Hormes]

Hi, I am having trouble cleaning up a javassist.ClassPool object. How is this done?