How to detect the cause of OutofMemoryError?

Question

I have a complaint that a server application of mine is crashing on high load.
It is a web app running in Tomcat 5.
I see the thread dumps and I see that there is an OutOfMemory error

1TISIGINFO Dump Event "systhrow" (00040000) Detail
"java/lang/OutOfMemoryError" "Failed to create thread: retVal -1073741830, errno 12" >received 1TIDATETIME Date: 2012/07/17 at 20:03:17 1TIFILENAME >Javacore filename:C:\ServerApplication\Tomcat5\bin\javacore.34545719.4646464.4172.0003.txt

The heap information is the following:

Maximum Java heap size : 1500m    
Initial Java heap size : 256m

This is the configuration for initial and Max heap size (32 bit java)

I also see that there is available free heap space

1STHEAPFREE    Bytes of Heap Space Free: 2D19F3C0   
1STHEAPALLOC   Bytes of Heap Space Allocated: 5DC00000

This is arround 750MB free space, right?

And from the thread method analysis I see that the number of Threads is 695 of which 49% is java/lang/Object.wait(Native Method) and 39% is in sun/misc/Unsafe.park(Native Method)
Also I see this NO JAVA STACK 1% not sure what that means.
Also 0 treads deadlocked and 2% is Runnable.

I am not sure how to interpret this information or how to go on from here to detect the underlying cause.
Any help on this?

Solution 1

According to this post:

There are two possible causes of the java.lang.OutOfMemoryError: Failed to create a thread message:

• There are too many threads running and the system has run out of internal resources to create new threads.
• The system has run out of native memory to use for the new thread. Threads require a native memory for internal JVM structures, a Java™ stack, and a native stack.

So this error may well be completely unrelated to memory, just that too many threads are created...

EDIT:

As you've got 695 threads, you would need 695 times the stack size as memory. Considering this post on thread limits, I suspect that you are trying to create too many threads for the available virtual memory space.

Solution 2

You should start the JVM with the -XX:+HeapDumpOnOutOfMemoryError flag. This will produce a heap dump when the OutOfMemoryError is generated.

Then, as @Steve said, you can use a tool like MAT to analyze the dump and see which objects are allocated, and who is keeping references to them. This usually will give you some insight on why your JVM is exhausting its memory.

Solution 3

I know what you mean, it can be confusing to find somewhere to begin.

Have a look at Eclipse Memory Analyzer (MAT). It will use JHat to dump a memory snapshot of your program into a file, which you can re-open and analyze.

The browser for this file outlines all the objects created by the program very neatly, and you can look into various levels to find if something is suspicious.

---

Appending my comments to answer...

Right when your executable webapp crashes, dump it to MAT. MAT will tell you what object is being created a bunch of times. If it's a custom object, and it often is, it's easy to find. If not, you can see its parent, amputate it, and dribble down from there (sorry for the graphic example, I'm not entirely focused on SO at the moment :).

Oh, and I forgot to mention, you can run the program several times under several conditions, and make a dump each time. Then, you can analyze each dump for the trend.

---

But in my case what should I use?I have a web app running in Tomcat

Sorry, missed this too. If I'm not mistaken, MAT dumps the JVM process, so as long as the VM is running on your box you can dump its process and see what's going on.

---

Another comment mutated into partial solution...

This is becoming more difficult than it actually is. Seriously, it's pretty easy, after you run MAT once or twice to get the hang of things. Run your app until the thing crashes. Dump it. Change something. Run, crash, dump. Repeat. Then, open the dumps in MAT, and compare what looks suspicious.

The trickiest part when I was learning this was finding the process ID to dump - which is still not too mind-numbing.

Solution 4

A similar message on IBM WebSphere shows this line

"Failed to create a thread: retVal"

as indicative of a native OOM which means some thread (of the process) is trying to request a large portion of memory on the heap.

The IBM link above has a series of steps - some of which are IBM specific. Have a look.

From a native memory usage perspective:

• Maximum Java heap settings
• JDBC driver
• JNI code or Native libraries
• garbage collection of unused classes. Ensurethat -Xnoclassgc is not set.
• Thread Pool settings (fixed size thread pools)
• Too many classloaders etc, but these are not very common.
• Number of classes/classloaders from javacores.

Another thing you could look at is the PermGenSpace - how large is it?

This link http://www.codingthearchitecture.com/2008/01/14/jvm_lies_the_outofmemory_myth.html suggests

Increasing the heap allocation actually exacerbates this problem! It decreases the headroom the compiler, and other native components, have to play with. So the solution to my problem was: 1.reduce the heap allocated to the JVM. 2. remove the memory leaks caused by native objects not being freed in a timely fashion.

Also have you configured a value in server.xml for maxThreads ? The default is 200 but your app seems to have 695 ?

View more solutions

Comments

[Jim]

Ok.I can try this as well.But I am already using a tool for analysis and actually the data of the OP actually are data the tool helped me find.I just am not sure how to use them to find the root cause

[Jim]

I see that jmap expects an executable to map the memory segments.But in my case what should I use?I have a web app running in Tomcat.

[Ben]

Made some comments but they're better appended to the answer.

[Jim]

The problem is that I can not reproduce this here.You are saying that I should reproduce this and then connect to the process to collect the dumps?

[Jim]

So these issues can only be solved dynamically?By reproducing the error? One can not find the issue "statically" by looking into the logs?

[Ben]

Long story short, yes. My personal experience has led me to believe that errors must be reproduced in order to debug, unless you're a superhero of uncommon variety.

[Jim]

Ok.Thanks.But is there any kind of information I can extract from the fact that 45% of threads are waiting and 50% are parked? How can this lead to an OutofMemoryError if there is free heap space?

[Ben]

No worries, good luck. That gets into another question...the comment UI here is getting fussy so I've got to cut it short. Sounds like a good candidate for a new question. This is a pretty large topic, anyway. :)

[Jim]

Yes I use a pool of 500 threads in server.xml. Is this number too big?

[JoseK]

you could try reducing - when you allow 500 request threads incoming into the system each will require it's own file handler etc which takes up memory. having said that - the no. of request threads is different from the 695 application threads within the process - beny23's answer is better