How to determine why is Java app slow

Question

We have an Java ERP type of application. Communication between server an client is via RMI. In peak hours there can be up to 250 users logged in and about 20 of them are working at the same time. This means that about 20 threads are live at any given time in peak hours. The server can run for hours without any problems, but all of a sudden response times get higher and higher. Response times can be in minutes.

We are running on Windows 2008 R2 with Sun's JDK 1.6.0_16. We have been using perfmon and Process Explorer to see what is going on. The only thing that we find odd is that when server starts to work slow, the number of handles java.exe process has opened is around 3500. I'm not saying that this is the acual problem.

I'm just curious if there are some guidelines I should follow to be able to pinpoint the problem. What tools should I use? ....

Solution 1

Can you access to the log configuration of this application.

If you can, you should change the log level to "DEBUG". Tracing the DEBUG logs of a request could give you a usefull information about the contention point.

If you can't, profiler tools are can help you :

• VisualVM (Free, and good product)
• Eclipse TPTP (Free, but more complicated than VisualVM)
• JProbe (not Free but very powerful. It is my favorite Java profiler, but it is expensive)

If the application has been developped with JMX control points, you can plug a JMX viewer to get informations...

If you want to stress the application to trigger the problem (if you want to verify whether it is a charge problem), you can use stress tools like JMeter

Solution 2

Sounds like the garbage collection cannot keep up and starts "halt-the-world" collecting for some reason.

Attach with jvisualvm in the JDK when starting and have a look at the collected data when the performance drops.