Java - shutting down on Out of Memory Error

Question

I've heard very contradictory things on how to best handle this, and am stuck with the following dilemma:

• an OOME brings down a thread, but not the whole application
• and I need to bring down the whole application but can't because the thread doesn't have any memory left

I've always understood best practice is let them go so the JVM can die because the JVM is in an inconsistent state at that point, but that doesn't seem to be working here.

Solution 1

OutOfMemoryError is just like any other error. If it escapes from Thread.run() it will cause thread to die. Nothing more. Also, when a thread dies, it is no longer a GC root, thus all references kept only by this thread are eligible for garbage collection. This means JVM is very likely to recover from OOME.

If you want to kill your JVM no matter what because you suspect it can be in an inconsistent state, add this to your java options:

-XX:OnOutOfMemoryError="kill -9 %p"

%p is the current Java process PID placeholder. The rest is self-explained.

Of course you can also try catching OutOfMemoryError and handling it somehow. But that's tricky.

Solution 2

In Java version 8u92 the VM arguments

• -XX:+ExitOnOutOfMemoryError
• -XX:+CrashOnOutOfMemoryError

were added, see the release notes.

ExitOnOutOfMemoryError
When you enable this option, the JVM exits on the first occurrence of an out-of-memory error. It can be used if you prefer restarting an instance of the JVM rather than handling out of memory errors.

CrashOnOutOfMemoryError
If this option is enabled, when an out-of-memory error occurs, the JVM crashes and produces text and binary crash files.

Enhancement Request: JDK-8138745 (parameter naming is wrong though JDK-8154713, ExitOnOutOfMemoryError instead of ExitOnOutOfMemory)

Solution 3

With version 8u92 there's now a JVM option in the Oracle JDK to make the JVM exit when an OutOfMemoryError occurs:

From the release notes:

ExitOnOutOfMemoryError - When you enable this option, the JVM exits on the first occurrence of an out-of-memory error. It can be used if you prefer restarting an instance of the JVM rather than handling out of memory errors.

Solution 4

You can force your program to terminate in multiple ways, once the error will ocurre. Like others have suggested, you can catch the error and do a System.exit after that, if needed. But I suggest you too use -XX:+HeapDumpOnOutOfMemoryError, this way the JVM will create a memory dump file with the content of your application once the event was produced. You will use a profiles, I recommend you Eclipse MAT to investigate the image. This way you will find pretty quickly what is the cause of the issue, and react properly. If you are not using Eclipse you can use the Eclipse MAT as a standalone product, see: http://wiki.eclipse.org/index.php/MemoryAnalyzer.

Solution 5

If you want to bring down your program, take a look at the -XX:OnOutOfMemoryError="<cmd args>;<cmd args>" (documented here) option on the command line. Just point it to a kill script for your application.

In general, I have never had any luck to gracefully handle this error without restarting the application. There was always some kind of corner case slipping through, so I personally suggest to indeed stop your application but investigate the source of the problem.

View more solutions

Comments

[mcoolive]

Agree about the memory dump to analyse the cause. Disagree about to hardcode System.exit. In my opinion this strategy is dangerous, so if we choose it, I prefer to use options on the command line as the integrator will see them and be less surprised (and he will be able to change it).

[simbo1905]

Warning: an abrupt kill -9 may take down the process before your logs have flushed so it may look like a crash with no indication of what happened. Would be wise to use a chain of commands which try a polite shutdown "stop.sh %p" where your stop script can log that it is about to kill the process then try "kill -TERM $1", then sleep, and then do kill -9 last. That way you won't experience mystery crashes when your jvm commits suicide without logging what it was doing just before in the main logs.

[Harald]

This is at least an attempt to leave a helpful trace. If, however, this code runs into another OOM, I wonder if you get an infite exception handler recursion loop.

[Shloim]

You can make sure it doesn't if you unset the default exception on the first line of the handler.

[radlan]

Unfortunately this only works for "real" OutOfMemoryExceptions where the heap is exhausted. If the error occurs because native threads are exhausted, the OnOutOfMemoryError won't be triggered. See bugs.openjdk.java.net/browse/JDK-8155004

[radlan]

Unfortunately this only works for "real" OutOfMemoryExceptions where the heap is exhausted. If the error occurs because native threads are exhausted, the ExitOnOutOfMemoryError won't be triggered. See bugs.openjdk.java.net/browse/JDK-8155004

[radlan]

Unfortunately this only works for "real" OutOfMemoryExceptions where the heap is exhausted. If the error occurs because native threads are exhausted, the ExitOnOutOfMemoryError and CrashOnOutOfMemoryError won't be triggered. See bugs.openjdk.java.net/browse/JDK-8155004

[Roland Weber]

If the native threads are exhausted, there's another problem: the JVM would have to fork() in order to execute the kill command, but that would require another native thread. I found this little tool that might help: github.com/airlift/jvmkill

[SusanW]

Careful here: this might work for simple apps, or maybe when there's one task which allocates huge objects and gets OOM'ed a lot. Otherwise, when you use any 3rd party frameworks, it will tend to be their threads that get hit by the OOME, not yours, and then the framework will be crippled, and your handler won't be called so your app will be zombified: not working, not exited. Generally it's not a reliable approach.