heapdump size vs hprof size

Question

I recently made a heapdump in a hprof format when my jboss server was running with a xms of 4096m and xmx of 4096m and a permsize of 512m.

The hprof file generated is over 5gb. When I load the heapdump in visualvm, mat analyzer or yourkit, I only see a total bytes of approximately 1gb. I've tried changed the reachability scope in yourkit but it does not show more than 1 gb.

Any idea what this big difference in filesize vs displayed heapdump size can cause?

ps: I'm using jdk1.6.0_23

Unfortunately I'm not allowed to submit screenshots here.

On the filesystem the hprof size is of 5.227.659 kb and in yourkit it states:

Objects: 9.738.282 / shallow size 740 mb / retained size: 740 mb String reachable among them: 6.652.515 (68%) / shallow size: 381 mb (51%) / retained size: 381 MB (51%)

The largest retained size is a byte[] of 206.810.176

Solution 1

Did you try "Unreachable Objects Histogram" (you can find the link from the top of "Overview" page)? In one of my heapdumps sized 1509MB, mat shows only 454MB, but the rest is essentially garbage, and sure enough, the sum of "Shallow Heap" in unreachable objects histogram is 966MB.

Solution 2

which command did you use to generate heap dump?

$JAVA_HOME/bin/jmap -dump:live,format=b,file=c:/tmp/heap_dump.bin PID

maybe you need to pass live option, according to spec

 -dump:<dump-options> to dump java heap in hprof binary format
                   dump-options:
                     live         dump only live objects; if not specified,
                                  all objects in the heap are dumped.

Solution 3

This just means that most likely your heap-dump consisted of a large amount of unreachable objects that would have been garbage collected, if a GC were to run. Now that does not mean that you don't still have a leak, it just means that in your 5 GB Hprof, 4 GB of objects were unreachable and hence were not interesting sources of a leak.

In Java a memory leak can only occur if Garbage Collection can't clean out an object because something is holding a reference to it (unexpectedly). So your leak (if any) is to be found in the 1 GB of objects that remained in your hprof.

Comments

[Michael]

Hi, we use the following command: jmap -F -dump:format=b,file=${filename} $PID . according to oracle the live option: " If specified, only the live objects in the heap are dumped." So if I specify this, the heapdump most likely will be smaller but might also decrease my chances of finding the cause of a memory leak

[Andrey Borisov]

this is really good question - what is live object? i assume that live object is something that is not available for GC. So i guess live option will just remove garbage objects and you will have only live objects - this does not mean that this decreases chances of finding... I did found few memory leaks recently with live options in really huge dump (about 16 GB)...

[Michael]

Hi Andrey, thanks for your response. "I did found few memory leaks recently with live options in really huge dump (about 16 GB)" I can imagine that it might help but it still doesn't make sense why there is such a big difference between the hprof size on disk and the size the analyzer tools show when I load the hprof (heapdump) in such tools...

[gfan]

How to get the value of specified type int Unreachable Objects Histogram ? for example, I find java.lang.String occupied 500M, but I can't find what string is. I only can get what type is and the total size of the type, but can't find out the content in Eclipse Memory Analyzer.