Java on Linux insufficient memory even though there is plenty of available memory being used for caching

Question

I’m trying to start a process on Red Hat Enterprise Linux Server release 6.5 (Santiago), which is failing because it is not able to allocate sufficient memory.

# There is insufficient memory for the Java Runtime Environment to continue. 
# Cannot create GC thread. Out of system resources.
# An error report file with more information is saved as:

The problem is not with the Java application itself - I get the error too when I simply run:

java -version

Looking at memory consumption:

free -m

             total       used       free     shared    buffers     cached
Mem:         32069      31276        792          0        556      16948
-/+ buffers/cache:      13771      18297
Swap:         9325         38       9287

So Mem is reporting that 31 out of 32 Gb of memory is being used, but there should be 18Gb of memory available that is being used by the cache. Should the OS not free up some of this when a process requests memory?

The details of the error dump is

#
# There is insufficient memory for the Java Runtime Environment to continue.
# Cannot create GC thread. Out of system resources.
# Possible reasons:
#   The system is out of physical RAM or swap space
#   In 32 bit mode, the process size limit was hit
# Possible solutions:
#   Reduce memory load on the system
#   Increase physical memory or swap space
#   Check if swap backing store is full
#   Use 64 bit Java on a 64 bit OS
#   Decrease Java heap size (-Xmx/-Xms)
#   Decrease number of Java threads
#   Decrease Java thread stack sizes (-Xss)
#   Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
#  Out of Memory Error (gcTaskThread.cpp:46), pid=40816, tid=140071992215296
#
# JRE version: 6.0_26-b03
# Java VM: Java HotSpot(TM) 64-Bit Server VM (20.1-b02 mixed mode linux-amd64 compressed oops)

---------------  T H R E A D  ---------------

Current thread (0x00007f6508006800):  JavaThread "Unknown thread" [_thread_in_vm, id=40817,              stack(0x00007f650d46c000,0x00007f650d56d000)]

Stack: [0x00007f650d46c000,0x00007f650d56d000],  sp=0x00007f650d56b7c0,  free space=1021k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x85ebd5]  VMError::report_and_die()+0x265
V  [libjvm.so+0x3e41b8]  report_vm_out_of_memory(char const*, int, unsigned long, char const*)+0x68
V  [libjvm.so+0x466ada]  GCTaskThread::GCTaskThread(GCTaskManager*, unsigned, unsigned)+0x13a
V  [libjvm.so+0x4655de]  GCTaskManager::initialize()+0x21e
V  [libjvm.so+0x465373]  GCTaskManager::GCTaskManager(unsigned)+0x13
V  [libjvm.so+0x722cad]  ParallelScavengeHeap::initialize()+0x4dd
V  [libjvm.so+0x836549]  Universe::initialize_heap()+0xa9
V  [libjvm.so+0x8360ea]  universe_init()+0x7a
V  [libjvm.so+0x4ac53b]  init_globals()+0x4b
V  [libjvm.so+0x81cc74]  Threads::create_vm(JavaVMInitArgs*, bool*)+0x214
V  [libjvm.so+0x51a7b0]  JNI_CreateJavaVM+0x80

Solution 1

Have a look into the jvm min max memory, do you use any web server? Look into the number of treads, memory settings for it- for Tomcat most of them are @server.xml. Look into Java Mission Control http://www.oracle.com/technetwork/java/javaseproducts/mission-control/java-mission-control-1998576.html to get extra insight. Linux usually keeps more memory to be ready when requested. For Java on Linux only something like JMC can help you get an idea on what's going on. Have a look in the standard output from the JVM.

Solution 2

Seems like the issue has nothing to do with memory, but with process limits. The user I'm running the process as had a really low process limit set to 1024.

ulimit -u
1024

and the thread count of all the proceses run by this user is close to that:

ps -eLf | grep 'myuser' | wc -l
1022

Comments

[notdazedbutconfused]

Nice link and good info for general Java memory issues, but I'm more curious here about why the OS does not deallocate cache memory for a simple "java -version".

[Alex H]

This is more related to GC you can have a look here cubrid.org/blog/dev-platform/… keeps the RAM ready, the JVM is requesting it and also handling it. Look at XX:PermSize and -XX:MaxPermSize and various memory settings

[notdazedbutconfused]

Thanks Alex, I think one of my team mates actually found the issue. Useful links nonetheless!

[Alex H]

Great, that's what matters anyhow ;)