JVM always beyond 100% CPU usage in CentOS 6

Question

Good afternoon. I'm having a really baffling issue with the Java VM on my CentOS 6 (64bit) server. It's a server with a quad-core CPU (with hyperthreading), 16GB of RAM and 2TB hard drive space (on software RAID1). I figured I'd ask here because, according to the FAQ, questions about "common tools" used by developers are appropriate. Hopefully someone can shed some light on the issue.

I run a piece of Java software on the server, 24h/day. Earlier today (detected a few hours after midnight on the server's locale, which is notable because it's the first day of the month) I received reports by users connecting to this software as clients about it suddenly having become unusable. The JVM was never interrupted or restarted. It had been last restarted only a few days ago, and it had been running normally since then (using about 5% or less CPU, which is normal).

This time, when I checked the process, it was essentially gobbling up all the CPU time it could wrestle from the other applications running on the server (as reported by top), or, way over 100% (as reported by ps). I tried stopping and restarting the application, but as soon as it was running, CPU usage would immediately shoot up to above 100% again. Memory usage is normal. I experimented with different JVM flags, to no avail. I tried downloading the latest version of the software and making a fresh install, but this didn't help either. Finally, I tried downloading another, completely distinct piece of Java software, but when I ran it on the server, it suffered from the same problem.

When I downloaded the software and ran it on my computer, CPU usage was normal. Therefore, I am assuming is wrong in the server, but I can't imagine what it could possibly be. The server's JVM is up to date, from a standard, repository package (no customization whatsoever).

From java -version:

java version "1.6.0_24"
OpenJDK Runtime Environment (IcedTea6 1.11.3) (rhel-1.48.1.11.3.el6_2-x86_64)
OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)

From uname -a:

2.6.32-131.21.1.el6.x86_64 #1 SMP Tue Nov 22 19:48:09 GMT 2011 x86_64 x86_64 x86_64 GNU/Linux

From centos-release/redhat-release/system-release:

CentOS release 6.2 (Final)

free reports about 14.5gb free RAM. df reports about 1.4tb free disk space in the /home partition (where the software in question and all their files were located).

Any ideas? Please ask if you need more data, and feel free to suggest procedures for me to try out/check. When looking up the problem, I found several instances of people having trouble with JVM stealing all the CPU on CentOS, but none of them seemed to match my problem.

Solution 1

You probably want to try the following commands

# service ntpd stop; date -s "`date`";service ntpd start;

I have CentOS6 with kernel 2.6.32-220.el6.x86_64 and jdk1.7.0_04, high CPU was observed and resolved by above command. no need of restart OS.

This appears to be a bug with the linux kernel as described at https://access.redhat.com/knowledge/solutions/154793. The above fix is described at http://blog.wpkg.org/2012/07/01/java-leap-second-bug-30-june-1-july-2012-fix/

Solution 2

I can also confirm the issue present in RHEL 6, Tomcat 7, and Sun/Oracle Java 7 & Java 6.

ntpd stop; date; ntpd start command worked.

uname:
2.6.32-220.2.1.el6.x86_64 #1 SMP Fri Dec 23 02:21:33 CST 2011 x86_64 x86_64 x86_64 GNU/Linux