Something keeps killing my Java process on Ubuntu, anyone know why?
Solution 1
Assuming the problem is the OOM killer, then it has killed your process in a desperate attempt to keep the OS functioning in a severe memory shortage crisis.
I would conclude that:
your JVM is actually using significantly more than 28Gb; i.e. you've got significant non-heap memory usage, and
the OS is not configured with an adequate amount of swap space.
I'd try adding more swap space, so that the OS can swap out parts of your application in an emergency.
Alternatively, reduce the JVM's heap size.
Note that "-Xmx ..." sets the maximum heap size, not the maximum amount of memory that your JVM can use. The JVM puts some stuff outside the heap, including such things as the memory for thread stacks and memory-mapped files that your application is using.
The syslog confirms that it is the OOM killer at work.
In what way does the linked syslog say so?
It says this:
Nov 15 13:53:49 ip-10-71-94-36 kernel: [3707038.606133] Out of memory: kill process 6368 (run.sh) score 4747288 or a child
Nov 15 13:53:49 ip-10-71-94-36 kernel: [3707038.606146] Killed process 9359 (java)
The console says that java was killed, not that it quit.
Correct. It was killed by the operating system's OOM killer.
If it had run out of memory it would typically throw an OutOfMemory exception, which it didn't.
That is what would have happened if you had filled up the Java heap.
That is not what is going on here. The actual problem is that there is not enough physical RAM to hold the Java heap. The OOM killer deals with it ...
I'm running with such a huge heap because I need to store millions of objects each of which require several kilobytes of RAM.
Unfortunately, you are trying to use way more RAM than is available on the system. This is causing virtual memory to thrash, affecting the entire operating system.
When the system starts to thrash badly, the OOM killer (not the JVM) identifies your Java process as the cause of the problem. It then kills it (with a SIGKILL) to protect the rest of the system. If it didn't, there is a risk that the entire system would lock up completely and need to be hard rebooted.
Finally, you said:
My box has 35.84 GB of RAM ...
That is rather a strange value. 32 GiB is 34,359,738,368 bytes or 34.35 GB.
But based on that and the observed behavior, I suspect that that is the available virtual memory rather than physical RAM. Alternatively, your "box" could be a virtual machine with RAM overcommit enabled at the hypervisor level.
Solution 2
Welcome to the OOM-killer, a linux 'feature' that is the bane of large-memory applications everywhere. There's no simple recipe to deal, just google for it and start reading and weaping.
While I can't put my mental fingers on a concise explanation of the shenigans of the OOM killer, I recall that the critical tuning parameter is called 'swappiness'. On one of our big servers, we have:
/etc/sysctl.conf:vm.swappiness=20
Read http://www.gentooexperimental.org/~patrick/weblog/archives/2009-11.html.
Solution 3
What JVM are you using? and what application server? It's possible that you're allocating way too much memory, and that can be problematic - the garbage collector might have trouble doing its job.
I'm not sure if this is your case, but I found quite interesting this article explaining the way Linux overcommits memory.
Solution 4
Ubuntu has a "watchdog" process which kills other processes when memory runs low. See the manpage: http://manpages.ubuntu.com/manpages/natty/man8/watchdog.8.html
Solution 5
wow, can you actually have 28 GB of heap?! May be you should try reducing it, keep it at no more than 50% of the RAM I think (so ~18 GB, or may be even 15 GB). Plus 171 Full GCs are a lot! How long was this app running? 171 in 2-3 days sounds huge. btw the gist indicates an OOM before termination - I think reducing the heap will fix it ( you may be limiting the JVM from expanding native space). Try adjusting various parameters, try stack size for example (-Xss) if needed. Check max perm size and other sections too. Its a memory problem and it may not necessarily be the heap.
sanity
I'm an entrepreneur and computer scientist, with a particular interest in Artificial Intelligence and Peer-to-Peer. My two most notable projects are Freenet and Revver (I'm founder and co-founder respectively). My current projects are a predictive analytics system called SenseArray, and a new approach to distributed computation called Swarm. You can find my personal blog here. While I've used C, C++, ML, Haskell, Prolog, Python, even Perl in the past, these days I do most of my programming in Java. I am gaining experience with Scala though and expect to become my primary language as it and its tools mature. I was honored to be asked by Scala's creator to be on the program committee for the first Scala workshop.
Updated on June 05, 2022Comments
-
sanity almost 2 years
So every couple of days my java process on Ubuntu is killed automatically, and I can't figure out why.
My box has 35.84 GB of RAM, when I launch my Java process I pass it the -Xmx28g parameter, so it should be using way less than the maximum RAM available.
I ran jstat as follows:
# jstat -gccause -t `pgrep java` 60000
The last few lines of output from jstat immediately before the process was killed were:
Time S0 S1 E O P YGC YGCT FGC FGCT GCT LGCC GCC 14236.1 99.98 0.00 69.80 99.40 49.88 1011 232.305 11 171.041 403.347 unknown GCCause No GC 14296.2 93.02 0.00 65.79 99.43 49.88 1015 233.000 11 171.041 404.041 unknown GCCause No GC 14356.1 79.20 0.00 80.50 99.55 49.88 1019 233.945 11 171.041 404.986 unknown GCCause No GC 14416.2 0.00 99.98 24.32 99.64 49.88 1024 234.945 11 171.041 405.987 unknown GCCause No GC
This seems to be what went down in the /var/log/syslog around this time: https://gist.github.com/1369135
There is really nothing running on this server other than my java app. What's going on?
edit: I'm running java version 1.6.0_20, the only notable parameters I'm passing to java on startup are "-server -Xmx28g". I'm not using an application server but my app embeds the "Simple web framework".
-
sanity over 12 yearsIn what way does the linked syslog say so? The console says that java was killed, not that it quit. If it had run out of memory it would typically throw an OutOfMemory exception, which it didn't. I'm running with such a huge heap because I need to store millions of objects each of which require several kilobytes of RAM.
-
aishwarya over 12 yearsoops misread, its actually 11 full GCs, but thats still quite a few. its definitely a OOM and you should try reducing the heap size.
-
sanity over 12 yearsSorry, I'm not seeing where the gist indicates an OOM, I can't see any OOM in stdout or stderr for the process :-/ Why is it important that the entire heap only use 50% of RAM? Surely the -Xmx specifies the maximum RAM Java will use?
-
sanity over 12 years..also, won't reducing the heap size increase the likelihood of an OOM?
-
sanity over 12 yearsUpdated my question with the info you requested
-
sanity over 12 yearsCan you summarize why the OOM-killer would kill an app using only 28GB on a machine with 35GB of RAM and basically no other processes running on it?
-
aishwarya over 12 yearsin the gist, the first line says that the jvm has invoked oom killer and the last but one line says that there was a out of memory :-) the 50% thing is more of a general guideline i use for myself rather than anything else, because the JVM needs space for natives (threads, primitives - basically runtime stacks), class loading and for its own operation. all of this is not accounted for in the heap. typically, the heap to other memory needs ratio is around 1:1 and hence the 50% rule (my own). OOM is also caused when the JVM is unable to create threads or load classes, so try reducing the heap
-
sanity over 12 yearsAh, apologies. When you said "OOM" I thought you meant a "Java OOM exception", not the OS complaining about an OOM
-
sanity over 12 yearsOh, I've reduced the heap to 15GB per your recommendation
-
sanity over 12 yearsPer some other advice I'm reducing the max memory usage to 15GB
-
yegor256 almost 11 years
-
V.Y. about 5 yearsThe link seems to be broken