Java process eating CPU; Why?

Question

I have a Linux server which I have installed Java on.

Sometimes, and only sometimes when a large number of visitors visit my website, the site hangs.

When I open the terminal and enter the "top" command to see whats going on, I can see that "Java" process is eating CPU! Like 400%.

I have also tried ps aux command, and can see that the command is from usr/bin/java

I have little experience in troubleshooting this kind of things, so I turn to you guys for help.

I have a java container installed (Jetty) which I must have in order to use SOLR (search engine) which is integrated into my website.

I can start and stop SOLR by:

  etc/init.d/solr stop

But this didn't remove the java process from the "Top" command. Still java was eating 400% CPU.

Is there other methods to restart java only?

This has happened twice to me, and each time I have now restarted my entire servers and everything is fine.

If you need more input let me know!

Solution 1

If you stop the solr process and a java process is still running, then there's another java process on your server. The first step is to document all of the Java processes that are running. A good tool for this on Unix is the ps tool. Try this:

$ ps auxwww | grep java

That output should show you all of the java processes running, and the commands that are being executed. Try this before and after you stop solr.

The second question is "which of these java processes is eating up so much CPU"? You may need to stop the jetty process in addition to the solr process. Also, you should never really need to restart a Linux server just because a single process is misbehaving. You can also use the kill command to stop a process if you now it's process id, which you can get from either top or ps.

In the short term it may be a good idea to install some sort of "watchdog" script on your machine to help with these situations. For example, monit can be used to automatically restart a service or process when it consumes a certain amount of CPU resources.

In the long term, I'm sorry to say that you have a performance issue. You need to look at reconfiguring solr and jetty at the very least. You may also need to look into garbage collection tuning and possibly adding more hardware. There's lots of information on these topics online, and I'm sorry to say that this process can be somewhat difficult.

Good luck!

Tom Purl

Solution 2

If the CPU is getting maxed out while response times are dropping, then the first place I'd be looking is the heap management within the java - it's probably doing lots of major collections - and the quickest way to solve the problem is to just give the container more memory to run in.

Comments

[Anonymous12345]

Broken configuration? I am asking how to troubleshoot here. Could you explain some tips?

[Admin]

See @Tom Purl's answer, which details what I'm referring to: "reconfiguring solr and jetty at the very least".

[Admin]

Nice answer. +1 for you.