Web application very slow in Tomcat 7

Question

I implemented a web application to start the Tomcat service works very quickly, but spending hours and when more users are entering is getting slow (up to 15 users approx.).

Checking RAM usage statistics (20%), CPU (25%)

Server Features:

• RAM 8GB
• Processor i7
• Windows Server 2008 64bit
• Tomcat 7
• MySql 5.0
• Struts2
• -Xms1024m
• -Xmx1024m
• PermGen = 1024
• MaxPernGen = 1024

I do not use Web server, we publish directly on Tomcat.

Entering midnight slowness is still maintained (only 1 user online)

The solution I have is to restart the Tomcat service and response time is again excellent.

Is there anyone who has experienced this issue? Any clue would be appreciated.

Solution 1

Not enough details provided. Need more information :(

Use htop or top to find memory and CPU usage per process & per thread.

CPU

A constant 25% CPU usage in a 4 cores system can indicate that a single-core application/thread is running 100% CPU on the only core it is able to use.

Which application is eating the CPU ?

Memory

20% memory is ~1.6GB. It is a bit more than I expect for an idle server running only tomcat + mysql. The -Xms1024 tells tomcat to preallocate 1GB memory so that explains it.

Change tomcat settings to -Xms512 and -Xmx2048. Watch tomcat memory usage while you throw some users at it. If it keeps growing until it reaches 2GB... then freezes, that can indicate a memory leak.

Disk

Use df -h to check disk usage. A full partition can make the issues you are experiencing.

Filesystem    Size  Used Avail Usage% Mounted on
/cygdrive/c     149G  149G  414M 100%   /

(If you just discovered in this example that my laptop is running out of space. You're doing it right :D)

Logs

Logs are awesome. Yet they have a bad habit to fill up the disk. Check logs disk usage. Are logs being written/erased/rotated properly when new users connect ? Does erasing logs fix the issue ? (copy them somewhere for future analysis before you erase them)

If not. Logs are STILL awesome. They have the good habit to help you track bugs. Check tomcat logs. You may want to set logging level to debug. What happens last when the website die ? Any useful error message ? Do user connections are still received and accepted by tomcat ?

Application

I suppose that the 25% CPU goes to tomcat (and not mysql). Tomcat doesn't fail by itself. The application running on it must be failing. Try removing the application from tomcat (you can eventually put an hello world instead). Can tomcat keep working overnight without your application ? It probably can, in which case the fault is on the application.

Enable full debug logging in your application and try to track the issue. Run it straight from eclipse in debug mode and throw users at it. Does it fail consistently in the same way ?

If yes, hit "pause" in the eclipse debugger and check what the application is doing. Look at the piece of code each thread is currently running + its call stack. Repeat that a few times. If there is a deadlock, an infinite loop, or similar, you can find it this way.

You will have found the issue by now if you are lucky. If not, you're unfortunate and it's a tricky bug that might be deep inside the application. That can get tricky to trace. Determination will lead to success. Good luck =)

Solution 2

I experienced a very slow stock Tomcat dashboard on a clean Centos7 install and found the following cause and solution:

Slow start up times for Tomcat are often related to Java's SecureRandom implementation. By default, it uses /dev/random as an entropy source. This can be slow as it uses system events to gather entropy (e.g. disk reads, key presses, etc). As the urandom manpage states:

When the entropy pool is empty, reads from /dev/random will block until additional environmental noise is gathered.

Source: https://www.digitalocean.com/community/questions/tomcat-8-5-9-restart-is-really-slow-on-my-centos-7-2-droplet

Fix it by adding the following configuration option to your tomcat.conf or (preferred) a custom file into /tomcat/conf/conf.d/:

JAVA_OPTS="-Djava.security.egd=file:/dev/./urandom"

Solution 3

For performance related issue, we need to follow the given rules:

1. You can equalize and emphasize the size of xms and xmx for effectiveness.

  -Xms2048m
  -Xmx2048m

2. You can also enable the PermGen to be garbage collected.

-XX:+UseConcMarkSweepGC -XX:+CMSPermGenSweepingEnabled -XX:+CMSClassUnloadingEnabled

3. If the page changes too frequently to make this option logical, try temporarily caching the dynamic content, so that it doesn't need to be regenerated over and over again. Any techniques you can use to cache work that's already been done instead of doing it again should be used - this is the key to achieving the best Tomcat performance.

4. If there any database related issue, then can follow sql query perfomance tuning

5. rotating the Catalina.out log file, without restarting Tomcat.

In details,There are two ways.

The first, which is more direct, is that you can rotate Catalina.out by adding a simple pipe to the log rotation tool of your choice in Catalina's startup shell script. This will look something like:

"$CATALINA_BASE"/logs/catalina.out WeaponOfChoice 2>&1 &

Simply replace "WeaponOfChoice" with your favorite log rotation tool.

The second way is less direct, but ultimately better. The best way to handle the rotation of Catalina.out is to make sure it never needs to rotate. Simply set the "swallowOutput" property to true for all Contexts in "server.xml".

This will route System.err and System.out to whatever Logging implementation you have configured, or JULI, if you haven't configured.

• See more at: Tomcat Catalina Out

Solution 4

We encountered a similar problem, the cause was "catalina.out". It is the standard destination log file for "System.out" and "System.err". It's size kept on increasing thus slowing things down and ultimately tomcat crashed. This problem was solved by rotating "catalina.out". We were using redhat so we made a shell script to rotate "catalina.out".

Here are some links:-

Mulesoft article on catalina (also contains two methods of rotating):

Tomcat Catalina Introduction

If "catalina.out" is not the problem then try this instead:-

Mulesoft article on optimizing tomcat: Tuning Tomcat Performance For Optimum Speed

View more solutions

Comments

[Mohammed Azharuddin Shaikh]

Tried with the JVM parameters(same result), FYI I'm using VM 8 cores, 64GB RAM, databse Hadoop Hbase.

[SkyWalker]

@ShaikhAzhar : So problem is not actually size related. So, I feel that the number 5 will be the best way. As you said, you need to restart, so we can avoid it by 5.

[SkyWalker]

@ShaikhAzhar: Could you please provide some database related info? that how much table is there and how much data are there in table? is the sql is well formed written? please check.

[Imaskar]

Our catalina.out sometimes grow to hundreds GB and still everything works fine. It just appends at the end after all.