Java OutOfMemory exception: mmap error on loading zip file

Question

I run my app on production env (rhel 5.2 x64, oracle jre 1.7_05, tomcat 7.0.28) with JVM arguments:

-Xms8192m -Xmx8192m -XX:MaxPermSize=1024m 
-Doracle.net.tns_admin=/var/ora_net -XX:ReservedCodeCacheSize=512m -XX:+AggressiveOpts -XX:+UseFastAccessorMethods 
-XX:+UseStringCache -XX:+OptimizeStringConcat -XX:+UseCompressedOops -XX:+UseG1GC -Dcom.sun.management.jmxremote 
-Dcom.sun.management.jmxremote.port=9026 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false

After several time i've got stack trace like that:

Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to allocate stack guard pages failed.
mmap failed for CEN and END part of zip file
[...]
Caused by: java.lang.OutOfMemoryError: null
    at java.util.zip.ZipFile.$$YJP$$open(Native Method) ~[na:1.7.0_05]
    at java.util.zip.ZipFile.open(Unknown Source) ~[na:1.7.0_05]
    at java.util.zip.ZipFile.<init>(Unknown Source) ~[na:1.7.0_05]
    at java.util.zip.ZipFile.<init>(Unknown Source) ~[na:1.7.0_05]
    at java.util.jar.JarFile.<init>(Unknown Source) ~[na:1.7.0_05]
    at java.util.jar.JarFile.<init>(Unknown Source) ~[na:1.7.0_05]
    at sun.net.www.protocol.jar.URLJarFile.<init>(Unknown Source) ~[na:1.7.0_05]
    at sun.net.www.protocol.jar.URLJarFile.getJarFile(Unknown Source) ~[na:1.7.0_05]
    at sun.net.www.protocol.jar.JarFileFactory.get(Unknown Source) ~[na:1.7.0_05]
    at sun.net.www.protocol.jar.JarURLConnection.connect(Unknown Source) ~[na:1.7.0_05]
    at sun.net.www.protocol.jar.JarURLConnection.getInputStream(Unknown Source) ~[na:1.7.0_05]
    at java.net.URL.openStream(Unknown Source) ~[na:1.7.0_05]
    at org.apache.catalina.loader.WebappClassLoader.findLoadedResource(WebappClassLoader.java:3279) ~[na:na]
    at org.apache.catalina.loader.WebappClassLoader.getResourceAsStream(WebappClassLoader.java:1478) ~[na:na]
    at org.apache.http.util.VersionInfo.loadVersionInfo(VersionInfo.java:242) ~[httpcore-4.2.jar:4.2]
    at org.apache.http.impl.client.DefaultHttpClient.setDefaultHttpParams(DefaultHttpClient.java:180) ~[httpclient-4.2.jar:4.2]
    at org.apache.http.impl.client.DefaultHttpClient.createHttpParams(DefaultHttpClient.java:158) ~[httpclient-4.2.jar:4.2]
    at org.apache.http.impl.client.AbstractHttpClient.getParams(AbstractHttpClient.java:448) ~[httpclient-4.2.jar:4.2]

Looking to my profiler - everthing is ok (heap and non-heap memory used for 10%) and i have no idea where is the problem.

This problem's happening every day at same time and it's not connected to application uptime. What is cause it problem?

Edited:

New output in log file:

Java HotSpot(TM) 64-Bit Server VM warning: CodeCache is full. Compiler has been disabled.
Java HotSpot(TM) 64-Bit Server VM warning: Try increasing the code cache size using -XX:ReservedCodeCacheSize=
Code Cache  [0x00002aaaab790000, 0x00002aaaad240000, 0x00002aaacb790000)
 total_blobs=4223 nmethods=3457 adapters=707 free_code_cache=497085Kb largest_free_block=508887936

But i have enough memory: http://i.stack.imgur.com/K8VMx.jpg

Answer: Problem in java version. It descripted here: https://forums.oracle.com/forums/thread.jspa?messageID=10369413

Solution

I have seen these error before when running out of resources such as running out of swap space or running out of allowed memory mapping. Have a look at sudo cat /proc/$PID/maps | wc -l compared with cat /proc/sys/vm/max_map_count

See comments below.

---

I also suggested ....

You appear to have run into a bug with YourKit. What version are you using?

I would cut down most of your options as they either are the default and don't do anything or could be complicating matters.

-mx8g -XX:MaxPermSize=1g -Doracle.net.tns_admin=/var/ora_net 
-XX:ReservedCodeCacheSize=512m -XX:+UseG1GC -Dcom.sun.management.jmxremote.port=9026

I would try dropping -XX:+UseG1GC as well as this is a relatively new collector and shouldn't change your results.

Comments

[Erik]

Out of curiosity, how do you figure the Yourkit part?

[Darya Dmitrichenko]

Yep, i'm using yjp-11.0.8, but the problem happens before i've instaleld it. I'll try to drop G1 collector, but i think it doesn't solve my problem because i work with G1 for the long time. Also i have a problem right now: yjp_out.

[Vishy]

I have seen these error before when running out of resources such as running out of swap space or running out of allowed memory mapping. Have a look at sudo cat /proc/$PID/maps | wc -l compared with cat /proc/sys/vm/max_map_count

[Darya Dmitrichenko]

Yes, I've got 65538/65536. It's problem :). I executed sysctl -w vm.max_map_count=100000 but i still need to restart my application to apply changes?

[Vishy]

I haven't tried this. Once you get the error I would think you should restart. BTW I would also look at minimising the number of mappings as 64K is allot.

[Darya Dmitrichenko]

There is a lot of deleted maps there: cat /proc/26464/maps |wc -l: 65538 cat /proc/26464/maps |grep "(deleted)" |wc -l: 64728 Like that: 2aab16919000-2aab1691a000 rw-s 00000000 00:14 25781321 /dev/shm/sem.YmuIm2 (deleted) Maybe the problem in deleted but not released maps?

[Vishy]

I suspect deleted maps cannot be released.

[Darya Dmitrichenko]

Thank you, your answers help me figure out the reason of my problem.