Apache Tomcat Request Threads

Question

We have an application which leaks a bit of memory, a bit being an understatement.

I am using jvisualvm to try and find what is causing the problem.

I see the thread count grow quite a bit on threads starting with the name: http-8080- example: http:8080-42

My first guess is that each of those threads is a request hit from the client, as each client request is handled in its own thread.

My my problem is that those threads have been running for long periods of time (Thus far 10mins).

My question is this:

Is my assumption correct? If so, why is it that the Threads run for such a long time? Surely it can't still be busy serving the clients request?

Solution 1

Tomcat always has a number of waiting HTTP threads, for example if we look at the default connector setting:

<Connector port="80" maxHttpHeaderSize="8192"
              maxThreads="150" minSpareThreads="25" maxSpareThreads="75"
              enableLookups="false" redirectPort="8443" acceptCount="100"
              connectionTimeout="20000" disableUploadTimeout="true" />

We can see that there should always be at least 25 threads live, but waiting for connections (up to the maxThreads limit). This is controlled by the min and maxSpareThreads attributes.

What does JVisual VM state that the thread is doing waiting or locked on a resource etc etc?

Solution 2

Generally speaking, application servers will pre create a number of threads. Not only will the app server create them, but it will keep the threads around. This is known as a thread pool. The server will take a request and dispatch it to a thread, and when that request completes, the server will dispatch a new request to that thread.

Thread creation overhead is rather expensive so handling many requests benefits greatly from sharing threads. To answer your question, dispatching threads created by the server (assuming no serious runtime errors occur) will live for the lifetime of the server.

As for what you are seeing, if you see many many threads being started, then some other part of the application can be forking threads in which is a completely separate issue.

Its important to know that your tomcat server should not be creating new threads for each request (again generally speaking) it should reusing threads.

Solution 3

Check the tomcat connector configurations. Pay attention to maxThreads and other thread pool configuration. A common mistake is to just increase maxThreads without actually "Tuning". If you configure an unnecessarily large pool, it will lead to lots of idle threads. This will do no good.

Even though it's obvious, just for the record, TIMED_WAITING threads will time out, and WAITING threads will just lay around for a notify() or a notifyAll().

Comments

[Koekiebox]

All of those threads are in a WAITING state: "http-8080-24" - Thread t@70 java.lang.Thread.State: WAITING at java.lang.Object.wait(Native Method) - waiting on <96c084> (a org.apache.tomcat.util.net.JIoEndpoint$Worker) at java.lang.Object.wait(Object.java:485) at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpo‌​int.java:414) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoin‌​t.java:440) at java.lang.Thread.run(Thread.java:619) Locked ownable synchronizers: - None

[Michael]

Yep, they're just awaiting more connections - perfectly expected (and desirable behaviour). :)