Jenkins java.lang.OutOfMemoryError: GC overhead limit exceeded

Question

I am currently working on creating a performance framework using jenkins and execute the performance test from Jenkins. I am using https://github.com/jmeter-maven-plugin/jmeter-maven-plugin this plugin. The sanity test with single user in this performance framework worked well and went ahead with an actual performance test of 200 users and within 2 mins received the error java.lang.OutOfMemoryError: GC overhead limit exceeded I tried the following in jenkins.xml

<arguments>-Xrs -Xmx2048m -XX:MaxPermSize=512m -Dhudson.lifecycle=hudson.lifecycle.WindowsServiceLifecycle -jar "%BASE%\jenkins.war" --httpPort=8080 --prefix=/jenkins --webroot="%BASE%\war"</arguments>

but it didn't work and also noted that whenever I increased the memory the jenkins service stops and had to reduce the memory to 1Gb and then the service restarts.

Had increased the memory for jmeter and java as well but no help. In the .jmx file view results tree and every other listener is disabled but still the issue persists.

Since I am doing a POC jenkins is hosted in my laptop and high level specs as follows System Model : Latitude E7270 Processor : Intel(R) Core(TM) i5-6300U CPU @ 2.40GHZ(4CPU's), ~2.5GHZ Memory : 8192MB RAM

Any help please ?

Solution

The error about GC overhead implies that Jenkins is thrashing in Garbage Collection. This means it's probably spending more time doing Garbage Collection than doing useful work.

This situation normally comes about when the heap is too small for the application. With modern multi generational heap layouts it's difficult to say what exactly needs changing.

I would suggest you enable Verbose GC with the following options "-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps"

Then follow the advice here: http://www.oracle.com/technetwork/articles/javase/gcportal-136937.html

Comments

[hardillb]

What is the error when you increase the memory (PermSize or heap?) to 2GB? Also what are the specs of the machine all this is running on?

[Prashanth K]

Since I am doing a POC jenkins is hosted in my laptop and its high level specs as follows System Model : Latitude E7270 Processor : Intel(R) Core(TM) i5-6300U CPU @ 2.40GHZ(4CPU's), ~2.5GHZ Memory : 8192MB RAM There is no error when I increase the memory to 2GB(Its only Heap I think) simply the jenkins service stops

[hardillb]

Edit the question to add that info and the include the information about which cmd line arg you changed to increase the memory to 2GB.

[Prashanth K]

Added the info and regarding the cmd line argument..its already posted in the question.

[Prashanth K]

I tried this <arguments>-Xrs -Xmx2048m -XX:MaxPermSize=512m -Dhudson.lifecycle=hudson.lifecycle.WindowsServiceLifecycle -jar "%BASE%\jenkins.war" --httpPort=8080 --prefix=/jenkins --webroot="%BASE%\war"</arguments> But Jenkins service and seeing this exception Error occurred during initialization of VM Could not reserve enough space for 2097152KB object heap Since 8gb of ram is available not sure why even 2GB is failing

[hardillb]

Are you running a 32bit JVM? add -d64 to the command line

[Prashanth K]

java version "1.8.0_112" Java(TM) SE Runtime Environment (build 1.8.0_112-b15) Java HotSpot(TM) 64-Bit Server VM (build 25.112-b15, mixed mode)

[Prashanth K]

"-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps" should this be included in jenkins.xml ?

[Prashanth K]

I am not using JSR223 sampler. regarding specifying or modifying new generation size do I need to specify in the jmeter I have installed in my laptop ? and I don't understand quite on the suggestion of moving test execution out of jenkins and provide the report as an input to plugin ? can you please elaborate on these aspects ?

[Selva]

When you use the jmeter maven plug-in, execution uses the jenkins JVM process and the jmeter classes and everything is loaded within jenkins. This is a poor model to scale since you are using jenkins as your controller host to run the test. You should add a jenkins slave (another windows machine with jmeter installed) and should run the test from the jenkins master to the slave. Jenkins master can connect to a slave and run windows batch script. Use jmeter command line interface to run your test and use CmdRunner to process your results.

[Selva]

And yes - all GC tuning that you do in your current model will be for the jenkins JVM since maven plug-in runs within the jenkins JVM. You should set the new generation on jenkins JVM just like how you have increased the heap size