Java : Get heap dump without jmap or without hanging the application

Question

In few circumstance, our application is using around 12 GB of memory. We tried to get the heap dump using jmap utility. Since the application is using some GB of memory it causes the application to stop responding and causes problem in production.

In our case the heap usage suddenly increases from 2-3 GB to 12GB in 6 hours. In an attempt to find teh memory usage trend we tried to collect the heap dump every one hour after restarting the application. But as said since using the jmap causes the application to hang we need to restart it and we are not able to get the trend of memory usage.

Is there a way to get the heap dump without hanging the application or is there a utility other than jmap to collect heap dump.

Thoughts on this highly appreciated, since without getting the trend of memory usage it is highly difficult to fix the issue.

Note: Our application runs in CentOS.

Thanks, Arun

Solution 1

Try the following. It comes with JDK >= 7:

/usr/lib/jvm/jdk-YOUR-VERSION/bin/jcmd PID GC.heap_dump FILE-PATH-TO-SAVE

Example:

/usr/lib/jvm/jdk1.8.0_91/bin/jcmd 25092 GC.heap_dump /opt/hd/3-19.11-jcmd.hprof

This dumping process is much faster than dumping with jmap! Dumpfiles are much smaller, but it's enough to give your the idea, where the leaks are.

At the time of writing this answer, there are bugs with Memory Analyzer and IBM HeapAnalyzer, that they cannot read dumpfiles from jmap (jdk8, big files). You can use Yourkit to read those files.

Solution 2

You can use GDB to get the heap dump without running jmap on the target VM however this will still hang the application for the amount of time required to write the heap dump to disk. Assuming a disk speed of 100MB/s (a basic mirrored array or single disk) this is still 2 minutes of downtime. http://blogs.atlassian.com/2013/03/so-you-want-your-jvms-heap/

The only true way to avoid stopping the JVM is transactional memory and a kernel that takes advantage of it to provide a process snapshot facility. This is one of the dreams of the proponents of STM but it's not available yet. VMWare's hot-migration comes close but depends on your allocation rate not exceeding network bandwidth and it doesn't save snapshots. Petition them to add it for you, it'd be a neat feature.

Solution 3

First of all, it is (AFAIK) essential to freeze the JVM while a thread dump / snapshot is being taken. If JVM was able to continue running while the snapshot was created, it would be next to impossible to get a coherent snapshot.

So are there other ways to get a heap dump?

• You can get a heap dump using VisualVM as described here.

• You can get a heap dump using jconsole or Eclipse Memory Analyser as described here.

But all of these are bound to cause the JVM to (at least) pause.

---

If your application is actually hanging (permanently!) that sounds like a problem with your application itself. My suggestion would be to see if you can track down that problem before looking for the storage leak.

My other suggestion is that you look at a single heap dump, and use the stats to figure out what kind(s) of object are using all of the space ... and why they are reachable. There is a good chance that you don't need the "trend" information at all.

Solution 4

A heap dump analyzed with the right tool will tell you exactly what is consuming the heap. It is the best tool for tracking down memory leaks. However, collecting a heap dump is slow let alone analyzing it.

With knowledge of the workings of your application, sometimes a histogram is enough to give you a clue of where to look for the problem. For example, if MyClass$Inner is at the top of the histogram and MyClass$Inner is only used in MyClass, then you know exactly which file to look for a problem.

Here's the command for collecting a histogram.

jcmdpidGC.class_histogram filename=histogram.txt

View more solutions

Comments

[Arun]

Thank Stephen. Are you suggesting to take histogram?

[Stephen C]

@Arun - I'm suggesting that you take a single heap dump, and use the available tools to 1) identify the largest or most common or most suspicious kinds of objects, and then 2) figure out how / why they are reachable. This is the normal approach to finding storage leaks in Java.