Why is my Java heap dump size much smaller than used memory?

Question

Problem

We are trying to find the culprit of a big memory leak in our web application. We have pretty limited experience with finding a memory leak, but we found out how to make a java heap dump using jmap and analyze it in Eclipse MAT.

However, with our application using 56/60GB memory, the heap dump is only 16GB in size and is even less in Eclipse MAT.

Context

Our server uses Wildfly 8.2.0 on Ubuntu 14.04 for our java application, whose process uses 95% of the available memory. When making the dump, our buffers/cache used space was at 56GB.

We used the following command to create the dump: sudo -u {application user} jmap -dump:file=/mnt/heapdump/dump_prd.bin {pid}

The heap dump file size is 16,4GB and when analyzing it with Eclipse MAT, it says there are around 1GB live objects and ~14,8GB unreachable/shallow heap.

EDIT: Here is some more info about the problem we see happening. We monitor our memory usage, and we see it grow and grow, until there is ~300mb free memory left. Then it stays around that amount of memory, until the process crashes, unfortunately without error in the application log.

This makes us assume it is a hard OOM error because this only happens when the memory is near-depleted. We use the settings -Xms25000m -Xmx40000m for our JVM.

Question

Basically, we are wondering why the majority of our memory isn't captured in this dump. The top retained size classes don't look too suspicious, so we are wondering if there is something heap dump-related what we are doing wrong.

Solution 1

When dumping its heap, the JVM will first run a garbage collection cycle to free any unreachable objects.

How can I take a heap dump on Java 5 without garbage collecting first?

In my experience, in a true OutOfMemoryError where your application is simply demanding more heap space than is available, this GC is a fool's errand and the final heap dump will be the size of the max. heap size.

When the heap dump is much smaller, that means the system was not truly out of memory, but perhaps had memory pressure. For example, there is the java.lang.OutOfMemoryError: GC overhead limit exceeded error, which means that the JVM may have been able to free enough memory to service some new allocation request, but it had to spend too much time collecting garbage.

It's also possible that you don't have a memory problem. What makes you think you do? You didn't mention anything about heap usage or an OutOfMemoryError. You've only mentioned the JVM's memory footprint on the operating system.

Solution 2

In my experience, having a heap dump much smaller than the real memory used can be due to a leak in the JNI.

Despite you don't use directly any native code, there are certain libraries that use it to speed up.

In our case, it was a Deflater and Inflater not properly ended.

Comments

[Thermometer]

Very nice explanation! i have updated the question with the answers to the things you ask. This could be the reason the heap dump is so much lower in size.

[Brice]

Unfortunately this is not exactly true. This only happens when the dump operation is used with the live sub-option : jmap -dump:live,file=... which in this case the there will be a full GC in order to dump only live objects. Which precisely did not happen there was both the command and mat indicate unreachable objects made it in the dump. There's several reasons why this could happen ful GC sometime before the dump, wide pointers...