Insufficient memory for Java Runtime Environment to continue in Tomcat

Question

I'm a tomcat/ tech newbie, so i'm sorry if I make any mistakes in the problem description.

I'm trying to run a dashboarding application, DOMO on our test server. We have been using tomcat to run the application since a decade, and everything usually works fine. Recently, when I tried to run the application, it wouldn't run. I cleared the work directory and tried to run tomcat again. Nothing worked and the application log for DOMO (the dashboarding tool) gave the following error:

Caused by: java.io.IOException: Insufficient system resources exist to complete the requested service
at java.io.WinNTFileSystem.canonicalize0(Native Method)
at java.io.Win32FileSystem.canonicalize(Unknown Source)
at java.io.File.getCanonicalPath(Unknown Source)
at org.apache.lucene.store.FSDirectory.getDirectory(FSDirectory.java:168)
at org.apache.lucene.store.FSDirectory.getDirectory(FSDirectory.java:139)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:951)
... 14 more

The system error log looked like this:

# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 32744 bytes for ChunkPool::allocate
# Possible reasons:
#   The system is out of physical RAM or swap space
#   In 32 bit mode, the process size limit was hit
# Possible solutions:
#   Reduce memory load on the system
#   Increase physical memory or swap space
#   Check if swap backing store is full
#   Use 64 bit Java on a 64 bit OS
#   Decrease Java heap size (-Xmx/-Xms)
#   Decrease number of Java threads
#   Decrease Java thread stack sizes (-Xss)
#   Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
#  Out of Memory Error (allocation.cpp:211), pid=3828, tid=2012
#
# JRE version: 6.0_31-b05
# Java VM: Java HotSpot(TM) 64-Bit Server VM (20.6-b01 mixed mode windows-amd64 compressed oops)

---------------  T H R E A D  ---------------

Current thread (0x000000000db09000):  JavaThread "C2 CompilerThread1" daemon [_thread_in_native, id=2012, stack(0x000000000dc80000,0x000000000dcc0000)]

Stack: [0x000000000dc80000,0x000000000dcc0000]

Current CompileTask:
C2:   8590 207  !   org.apache.catalina.loader.WebappClassLoader.findResourceInternal(Ljava/lang/String;Ljava/lang/String;)Lorg/apache/catalina/loader/ResourceEntry; (1260 bytes)


---------------  P R O C E S S  ---------------

Java Threads: ( => current thread )
  0x000000000ddee800 JavaThread "Thread-5" daemon [_thread_in_native, id=1272, stack(0x000000000fba0000,0x000000000fbe0000)]
  0x000000000e843000 JavaThread "Thread-4" daemon [_thread_blocked, id=2396, stack(0x000000000fb60000,0x000000000fba0000)]
  0x000000000e5dd000 JavaThread "HSQLDB Timer @3af7345b" daemon [_thread_blocked, id=1904, stack(0x000000000fb20000,0x000000000fb60000)]
  0x000000000e980800 JavaThread "GC Daemon" daemon [_thread_blocked, id=1564, stack(0x000000000f990000,0x000000000f9d0000)]
  0x000000000db51800 JavaThread "Thread-2" [_thread_in_native, id=1232, stack(0x000000000dd40000,0x000000000dd80000)]
  0x000000000db15000 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=1520, stack(0x000000000dcc0000,0x000000000dd00000)]
=>0x000000000db09000 JavaThread "C2 CompilerThread1" daemon [_thread_in_native, id=2012, stack(0x000000000dc80000,0x000000000dcc0000)]
  0x000000000daf1800 JavaThread "C2 CompilerThread0" daemon [_thread_blocked, id=4880, stack(0x000000000dc40000,0x000000000dc80000)]
  0x000000000daec000 JavaThread "Attach Listener" daemon [_thread_blocked, id=3120, stack(0x000000000d9e0000,0x000000000da20000)]
  0x000000000dae7000 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=4668, stack(0x000000000d9a0000,0x000000000d9e0000)]
  0x000000000dae5000 JavaThread "Surrogate Locker Thread (Concurrent GC)" daemon [_thread_blocked, id=4304, stack(0x000000000d960000,0x000000000d9a0000)]
  0x000000000da92000 JavaThread "Finalizer" daemon [_thread_blocked, id=2856, stack(0x000000000d920000,0x000000000d960000)]
  0x000000000da8b800 JavaThread "Reference Handler" daemon [_thread_blocked, id=1628, stack(0x000000000d8e0000,0x000000000d920000)]
  0x00000000002bc800 JavaThread "main" [_thread_in_native, id=3176, stack(0x00000000003c0000,0x0000000000400000)]

Other Threads:
  0x000000000da83800 VMThread [stack: 0x000000000d8a0000,0x000000000d8e0000] [id=3136]
  0x000000000db1f000 WatcherThread [stack: 0x000000000dd00000,0x000000000dd40000] [id=5084]

VM state:not at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread: None

Heap
 par new generation   total 19136K, used 16270K [0x000000069e400000, 0x000000069f8c0000, 0x00000006a0d90000)
  eden space 17024K,  83% used [0x000000069e400000, 0x000000069f1d3bf8, 0x000000069f4a0000)
  from space 2112K, 100% used [0x000000069f6b0000, 0x000000069f8c0000, 0x000000069f8c0000)
  to   space 2112K,   0% used [0x000000069f4a0000, 0x000000069f4a0000, 0x000000069f6b0000)
 concurrent mark-sweep generation total 2075904K, used 3163K [0x00000006a0d90000, 0x000000071f8d0000, 0x00000007f6000000)
 concurrent-mark-sweep perm gen total 21440K, used 21277K [0x00000007f6000000, 0x00000007f74f0000, 0x0000000800000000)

Code Cache  [0x0000000000c90000, 0x0000000000f00000, 0x0000000003c90000)
 total_blobs=526 nmethods=224 adapters=255 free_code_cache=49072960 largest_free_block=16512

Dynamic libraries:
0x0000000000400000 - 0x0000000000416000     c:\Tomcat-6.0.26\bin\tomcat6.exe
0x0000000077080000 - 0x0000000077229000     C:\Windows\SYSTEM32\ntdll.dll
0x0000000076f60000 - 0x000000007707f000     C:\Windows\system32\kernel32.dll
0x000007fefcf30000 - 0x000007fefcf9b000     C:\Windows\system32\KERNELBASE.dll
0x0000000076e60000 - 0x0000000076f5a000     C:\Windows\system32\USER32.dll
0x000007fefe350000 - 0x000007fefe3b7000     C:\Windows\system32\GDI32.dll
0x000007fefd680000 - 0x000007fefd68e000     C:\Windows\system32\LPK.dll
0x000007fefd6f0000 - 0x000007fefd7b9000     C:\Windows\system32\USP10.dll
0x000007fefe2b0000 - 0x000007fefe34f000     C:\Windows\system32\msvcrt.dll
0x000007fefe520000 - 0x000007fefe5fb000     C:\Windows\system32\ADVAPI32.dll
0x000007fefe110000 - 0x000007fefe12f000     C:\Windows\SYSTEM32\sechost.dll
0x000007fefd7c0000 - 0x000007fefd8ed000     C:\Windows\system32\RPCRT4.dll
0x000007fefe600000 - 0x000007feff388000     C:\Windows\system32\SHELL32.dll
0x000007fefdfc0000 - 0x000007fefe031000     C:\Windows\system32\SHLWAPI.dll
0x000007fefdad0000 - 0x000007fefdafe000     C:\Windows\system32\IMM32.DLL
0x000007fefd330000 - 0x000007fefd439000     C:\Windows\system32\MSCTF.dll
0x000000006d800000 - 0x000000006dfb8000     C:\Program Files\Java\jre6\bin\server\jvm.dll
0x000007fef9f50000 - 0x000007fef9f8b000     C:\Windows\system32\WINMM.dll
0x000000006d770000 - 0x000000006d77e000     C:\Program Files\Java\jre6\bin\verify.dll
0x000000006d3b0000 - 0x000000006d3d7000     C:\Program Files\Java\jre6\bin\java.dll
0x000000006d7c0000 - 0x000000006d7d2000     C:\Program Files\Java\jre6\bin\zip.dll
0x000007fefdb20000 - 0x000007fefdd23000     C:\Windows\system32\ole32.dll
0x000000006d550000 - 0x000000006d55a000     C:\Program Files\Java\jre6\bin\management.dll
0x0000000010000000 - 0x0000000010124000     C:\Tomcat-6.0.26\bin\tcnative-1.dll
0x0000000077240000 - 0x0000000077247000     C:\Windows\system32\PSAPI.DLL
0x000007fefe0c0000 - 0x000007fefe10d000     C:\Windows\system32\WS2_32.dll
0x000007fefd440000 - 0x000007fefd448000     C:\Windows\system32\NSI.dll
0x000007fefc660000 - 0x000007fefc6b5000     C:\Windows\system32\MSWSOCK.dll
0x000007fefc7f0000 - 0x000007fefc807000     C:\Windows\system32\CRYPTSP.dll
0x000007fefc3d0000 - 0x000007fefc417000     C:\Windows\system32\rsaenh.dll
0x000007fefcd50000 - 0x000007fefcd5f000     C:\Windows\system32\CRYPTBASE.dll
0x000007fefc070000 - 0x000007fefc077000     C:\Windows\System32\wshtcpip.dll
0x000000006d610000 - 0x000000006d627000     C:\Program Files\Java\jre6\bin\net.dll
0x000007fefc7e0000 - 0x000007fefc7e7000     C:\Windows\System32\wship6.dll
0x000007fefbbd0000 - 0x000007fefbbe5000     C:\Windows\system32\NLAapi.dll
0x000007fef8a90000 - 0x000007fef8aa5000     C:\Windows\system32\napinsp.dll
0x000007fefc4f0000 - 0x000007fefc54b000     C:\Windows\system32\DNSAPI.dll
0x000007fef8a80000 - 0x000007fef8a8b000     C:\Windows\System32\winrnr.dll
0x000007fefb2e0000 - 0x000007fefb307000     C:\Windows\system32\IPHLPAPI.DLL
0x000007fefb2d0000 - 0x000007fefb2db000     C:\Windows\system32\WINNSI.DLL
0x000007fef8e60000 - 0x000007fef8e68000     C:\Windows\system32\rasadhlp.dll
0x000007fefb140000 - 0x000007fefb193000     C:\Windows\System32\fwpuclnt.dll
0x000007fefce60000 - 0x000007fefce6f000     C:\Windows\system32\profapi.dll

VM Arguments:
jvm_args: -Dcatalina.base=c:\Tomcat-6.0.26 -Dcatalina.home=c:\Tomcat-6.0.26 -Djava.endorsed.dirs=c:\Tomcat-6.0.26\endorsed -Djava.io.tmpdir=c:\Tomcat-6.0.26\temp -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=c:\Tomcat-6.0.26\conf\logging.properties -Djava.security.auth.login.config=C:/adconf/bscLogin.conf -Djava.security.krb5.conf=C:/adconf/krb5.ini -XX:MaxPermSize=160m -XX:+UseConcMarkSweepGC -XX:+DisableExplicitGC -Dsun.rmi.dgc.client.gcInterval=360000 -Dsun.rmi.dgc.server.gcInterval=360000 vfprintf -Xms2048m -Xmx5500m 
java_command: <unknown>
Launcher Type: generic

Environment Variables:
JAVA_HOME=C:\Program Files\Java\jdk1.6.0_31\bin
CLASSPATH=C:\Domo\CenterView5\Server\domo\WEB-INF\lib\sqljdbc4.jar
PATH=C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\WindowsPowerShell\v1.0\;
USERNAME=BIDASH-DEV-APP$
OS=Windows_NT
PROCESSOR_IDENTIFIER=Intel64 Family 6 Model 23 Stepping 6, GenuineIntel



---------------  S Y S T E M  ---------------

OS: Windows NT 6.1 , 64 bit Build 7601 Service Pack 1

CPU:total 2 (1 cores per cpu, 1 threads per core) family 6 model 23 stepping 6, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1

Memory: 4k page, physical 8388152k(2153192k free), swap 33550744k(5136k free)

vm_info: Java HotSpot(TM) 64-Bit Server VM (20.6-b01) for windows-amd64 JRE (1.6.0_31-b05), built on Feb  3 2012 18:34:57 by "java_re" with MS VC++ 8.0 (VS2005)

time: Mon Jun 01 09:44:40 2015
elapsed time: 8 seconds

I've read a lot of posts on related topics but since I have had an application running for the past decade, this issue popping up all of a sudden seems really bizarre to me. Also, this environment is like a test server and currently I'm the only one using it for dashboards. Hence, the possibility of system load having increased doesn't seem right; nor the possibility of a memory leak.

Can anyone guide me as to what might have gone wrong in this situation?

Thanks!

EDIT: I think my problem is different since its not some new chuck of memory to be allocated, but a ChuckPool:: allocate operation which suddenly stopped working even though no new load was introduced on the server.

Solution

What is the min and max heap size for the JVM? The out of memory error occurred due to low memory for JVM heap. What is the OS having Tomcat? Is it a 32-bit OS? Also, what is the RAM memory on the system?

32-bit OS with let's say 4GB or higher RAM, maximum JVM heap size could only be 1.5GB. So, adjusting your max and min heap memory size to 1.5 GB could alleviate the problem. Since you are sure there is no memory leak, you should look at choosing right kind of GC collector for your application -a throughput based GC or other (parallel, etc) and optimize pause times and collections. This would allow JVM to have enough room for new object allocations and not run out of memory.