Java crashed application - how to find out why Java crashed?

java crash jvm out-of-memory jvm-crash

13,868

From the crash doc you linked, the error is a SIGSEGV which is a fault reading/writing to native memory. The thread stack shows it crashed in JVM code.

Current thread (0x00007fe899755800):  JavaThread "508616253@qtp-1871151428-3352" [_thread_in_vm, id=11941, stack(0x00007fe86a4e5000,0x00007fe86a5e6000)]

siginfo:si_signo=SIGSEGV: si_errno=0, si_code=128 (), si_addr=0x0000000000000000

Registers:
RAX=0x00007fe9c60333b8, RBX=0x00007fe899755800, RCX=0x0d00007fe8f58787, RDX=0x00007fe9c6031888
RSP=0x00007fe86a5e3fd0, RBP=0x00007fe86a5e4020, RSI=0x00007fe899755800, RDI=0x00007fe95bae1770
R8 =0x00007fe9be341620, R9 =0x0000000000000001, R10=0x00007fe9c5b84460, R11=0x00007fe9c051a52b
R12=0x00007fe9c051a529, R13=0x00007fe9c6034ac0, R14=0x00007fe9c051a599, R15=0x0900007fe8f58787
RIP=0x00007fe9c5bd562d, EFL=0x0000000000010246, CSGSFS=0x000000000000e033, ERR=0x0000000000000000
  TRAPNO=0x000000000000000d

Stack: [0x00007fe86a4e5000,0x00007fe86a5e6000],  sp=0x00007fe86a5e3fd0,  free space=3fb0000000000000030k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x64d62d]
V  [libjvm.so+0x5fc4df]

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
v  ~RuntimeStub::_complete_monitor_locking_Java
J  sun.nio.ch.SocketChannelImpl.write(Ljava/nio/ByteBuffer;)I
J  org.mortbay.io.nio.ChannelEndPoint.flush(Lorg/mortbay/io/Buffer;)I
J  org.mortbay.jetty.HttpGenerator.flush()J
<snip>

Could be a JVM bug, or perhaps memory corruption.

13,868

Libor Havlicek

Updated on May 20, 2022

Comments

Libor Havlicek almost 2 years

My java server started to crash repeatedly, and I can't find why.

I have server with 7.5GB memory and I have allocated 3GB for the java process.

Server was running fine, and ran garbage collection many times, but the JVM crashed when under memory pressure.

Here is the info from JConsole, after the crash:

Current heap size: 
2 958 868 kbytes
Maximum heap size: 
3 066 816 kbytes
Committed memory: 
3 066 816 kbytes
Pending finalization: 
0 objects
Garbage collector: 
Name = 'PS MarkSweep', Collections = 66, Total time spent = 7 minutes
Garbage collector: 
Name = 'PS Scavenge', Collections = 43 055, Total time spent = 44 minutes



Operating System: 
Linux 2.6.31-302-ec2
Architecture: 
amd64
Number of processors: 
2
Committed virtual memory: 
8 405 760 kbytes
Total physical memory: 
7 882 780 kbytes
Free physical memory: 
   34 540 kbytes
Total swap space: 
        0 kbytes
Free swap space: 
        0 kbytes

I have 0.5 GB after a GC run, so all the time it raises from 0.5 to 3 GB and than fall back to 0.5, it is absolutely not problem with hanging objects. In fact it should throw OutOfMemoryException instead of crashing. I am using those parameters:

-Xmn256m -Xms768m -Xmx3000m -XX:NewRatio=2 -server -verbosegc -XX:PermSize=256m -XX:MaxPermSize=256m -XX:SurvivorRatio=8 -XX:+UseParallelGC -XX:ParallelGCThreads=2 -XX:+UseParallelOldGC

What is wrong and what shall I do? The output shown was:

Current thread (0x00007fe899755800):  JavaThread "508616253@qtp-1871151428-3352" [_thread_in_vm, id=11941, stack(0x00007fe86a4e5000,0x00007fe86a5e6000)]

siginfo:si_signo=SIGSEGV: si_errno=0, si_code=128 (), si_addr=0x0000000000000000

Registers:
RAX=0x00007fe9c60333b8, RBX=0x00007fe899755800, RCX=0x0d00007fe8f58787, RDX=0x00007fe9c6031888
RSP=0x00007fe86a5e3fd0, RBP=0x00007fe86a5e4020, RSI=0x00007fe899755800, RDI=0x00007fe95bae1770
R8 =0x00007fe9be341620, R9 =0x0000000000000001, R10=0x00007fe9c5b84460, R11=0x00007fe9c051a52b
R12=0x00007fe9c051a529, R13=0x00007fe9c6034ac0, R14=0x00007fe9c051a599, R15=0x0900007fe8f58787
RIP=0x00007fe9c5bd562d, EFL=0x0000000000010246, CSGSFS=0x000000000000e033, ERR=0x0000000000000000
  TRAPNO=0x000000000000000d

Stack: [0x00007fe86a4e5000,0x00007fe86a5e6000],  sp=0x00007fe86a5e3fd0,  free space=3fb0000000000000030k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x64d62d]
V  [libjvm.so+0x5fc4df]

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
v  ~RuntimeStub::_complete_monitor_locking_Java
J  sun.nio.ch.SocketChannelImpl.write(Ljava/nio/ByteBuffer;)I
J  org.mortbay.io.nio.ChannelEndPoint.flush(Lorg/mortbay/io/Buffer;)I
J  org.mortbay.jetty.HttpGenerator.flush()J
...

Alexander Pogrebnyak about 13 years

Are there any JNI components?

andersoj about 13 years

And then switch to jmap and jhat to help diagnose the leak.
Libor Havlicek about 13 years

I am using amazon server, fresh 1 day old machine, since old virtual machine was defective, I hoped that new fresh will help. It is absolutly not hardware problem. It can be only linux problem, if it is not java problem.
Libor Havlicek about 13 years

Nope, it is garbage to be cleaned. I have all the time 0,5 gb after GB run, so all the time it raises from 0,5 to 3 GB and than fall back to 0,5, it is absolutly not problem with hanging objects. In fact it suould throw OutOfMemoryException instead of crash and no Exception. I am using those params: -Xmn256m -Xms768m -Xmx3000m -XX:NewRatio=2 -server -verbosegc -XX:PermSize=256m -XX:MaxPermSize=256m -XX:SurvivorRatio=8 -XX:+UseParallelGC -XX:ParallelGCThreads=2 -XX:+UseParallelOldGC
Libor Havlicek about 13 years

I alocated only 3GB, and I have 4,5Gb aditional free to use, so I have 7,5GB total. It is free momemory, no other big proceses running on the linux.
matbrgz about 13 years

Good, then that is clarified. Look into jvisualvm in the JDK - it allows you to profile a running application - you need to look for memory leaks.
matbrgz about 13 years

I will strongly recommend against tinkering with the JVM parameters until you have a reliable configuration working! Also this information is so important that it should go in your question.
Libor Havlicek about 13 years

And also I used to JAVA 5, JAVA 6 latest one, OpenJDK latest one, always same result. And JVM has generated crash files, not everytime, but sometimes yes. You can find example of crash file here: chessfriends-release.s3.amazonaws.com/logs/…
Vishy about 13 years

I found Java 6 update 18 to be stable. But its still worth trying update 23 or 24 which uses a much newer JVM. (more than a year AFAIK)
Libor Havlicek about 13 years

I used 6.23 also, same result.
Vishy about 13 years

The only simple thing I can suggest is trying the 32-bit version (though this might not have enough memory) It looks like a JVM/native bug to me. The next thing you could do is try Ubuntu 10.x :|
Vishy about 13 years

Can you try taking off as many GC options as possible. e.g. just -mx3g IMHO Sometimes certain combinations don't work well together and when I reduce them, the JVM behaves better.