How to interpret a Java thread stack?

java multithreading debugging glassfish threadpool

37,827

Solution 1

Couple of things that might help you

Here are the different thread states explained like BLOCKED, WAITING and etc.
Java will find dead locks for you and you will see output like described in Detecting Java Thread Deadlocks with 'jstack'.
You probably have a live lock. Here you need information on your system and going through the dump and the source code.

How to move forward with current problem?

Based on this dump I can tell you there are couple of objects that are locked and waited on from multiple threads. Look for strings '- waiting on' and '- locked'. I don't know the logic of your app to tell much more.

Solution 2

From your thread dump:

java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)

looks like you have the well known BUG in JDK with the Selector spinning.

37,827

Author by

Allan Lykke Christensen

Software Craftsman

Updated on August 25, 2020

Comments

Allan Lykke Christensen over 3 years

In line with this question: How to get stack trace of a thread

I am wondering if anyone could point to information about interpreting Java thread stacks extracted with jstack.

My situation is that I've got a Java EE 5 application running on GlassFish v2.1.1 that hangs periodically (at least 2 -3 times a day).To get it running again I have to kill the Glassfish process and start the domain again. The application usually gets slower and slower responding until it finally hangs completely. Once it has hung I can't get a thread stack. I've been able to get a thread stack as it gets slower and slower but I can't see to locate the problem or the threads that have hung.

Here is the latest thread dump: http://issues.i2m.dk/attachments/66/threadstack.201103211046.txt

Snippet of thread dump:

"p: thread-pool-1; w: 20" daemon prio=10 tid=0x00007efd18843800 nid=0x5f89 in Object.wait() [0x00007efd03381000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00007efd480fc228> (a com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl)
    at com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl.requestWork(WorkQueueImpl.java:171)
    - locked <0x00007efd480fc228> (a com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl)
    at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:528)

"TP-Processor83" daemon prio=10 tid=0x0000000043179000 nid=0x5f87 runnable [0x00007efd03583000]
   java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
    at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
    - locked <0x00007efe72696090> (a java.io.BufferedInputStream)
    at org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:607)
    at org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:545)
    at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:672)
    at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:876)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
    at java.lang.Thread.run(Thread.java:662)

"p: thread-pool-1; w: 19" daemon prio=10 tid=0x00007efd18710800 nid=0x5f86 in Object.wait() [0x00007efd03684000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00007efd480fc228> (a com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl)
    at com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl.requestWork(WorkQueueImpl.java:171)
    - locked <0x00007efd480fc228> (a com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl)
    at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:528)

"p: thread-pool-1; w: 18" daemon prio=10 tid=0x0000000042a5a000 nid=0x5f85 in Object.wait() [0x00007efd03785000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00007efd480fc228> (a com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl)
    at com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl.requestWork(WorkQueueImpl.java:171)
    - locked <0x00007efd480fc228> (a com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl)
    at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:528)

"p: thread-pool-1; w: 17" daemon prio=10 tid=0x0000000042a57000 nid=0x5f84 in Object.wait() [0x00007efd03886000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00007efd480fc228> (a com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl)
    at com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl.requestWork(WorkQueueImpl.java:171)
    - locked <0x00007efd480fc228> (a com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl)
    at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:528)

"p: thread-pool-1; w: 16" daemon prio=10 tid=0x000000004321b000 nid=0x5f83 in Object.wait() [0x00007efd03987000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00007efd480fc228> (a com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl)
    at com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl.requestWork(WorkQueueImpl.java:171)
    - locked <0x00007efd480fc228> (a com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl)
    at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:528)

"p: thread-pool-1; w: 15" daemon prio=10 tid=0x0000000043219800 nid=0x5f82 in Object.wait() [0x00007efd03a88000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00007efd480fc228> (a com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl)
    at com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl.requestWork(WorkQueueImpl.java:171)
    - locked <0x00007efd480fc228> (a com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl)
    at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:528)

"p: thread-pool-1; w: 14" daemon prio=10 tid=0x00000000429b6000 nid=0x5f81 in Object.wait() [0x00007efd03b89000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00007efd480fc228> (a com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl)
    at com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl.requestWork(WorkQueueImpl.java:171)
    - locked <0x00007efd480fc228> (a com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl)
    at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:528)

"TP-Processor82" daemon prio=10 tid=0x0000000043896800 nid=0x5f80 runnable [0x00007efd03c8a000]
   java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
    at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
    - locked <0x00007efe72829510> (a java.io.BufferedInputStream)
    at org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:607)
    at org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:545)
    at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:672)
    at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:876)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
    at java.lang.Thread.run(Thread.java:662)

"Thread-44980" daemon prio=10 tid=0x00007efd202c3800 nid=0x5d74 runnable [0x00007efd067b5000]
   java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
    at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
    - locked <0x00007efd78fc0058> (a java.io.BufferedInputStream)
    at com.sun.jndi.ldap.Connection.run(Connection.java:808)
    at java.lang.Thread.run(Thread.java:662)

"Thread-44979" daemon prio=10 tid=0x00007efd20d76000 nid=0x5d73 runnable [0x00007efd062b0000]
   java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
    at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
    - locked <0x00007efd7789a4e0> (a java.io.BufferedInputStream)
    at com.sun.jndi.ldap.Connection.run(Connection.java:808)
    at java.lang.Thread.run(Thread.java:662)

"TP-Monitor" daemon prio=10 tid=0x00007efd2139f800 nid=0x5b1 in Object.wait() [0x00007efd0d453000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00007efd4d46d6c0> (a org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
    at org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable.run(ThreadPool.java:561)
    - locked <0x00007efd4d46d6c0> (a org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
    at java.lang.Thread.run(Thread.java:662)

"TP-Monitor" daemon prio=10 tid=0x00007efd20143000 nid=0x5b0 in Object.wait() [0x00007efd0d554000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00007efd4d96dcd8> (a org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
    at org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable.run(ThreadPool.java:561)
    - locked <0x00007efd4d96dcd8> (a org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
    at java.lang.Thread.run(Thread.java:662)

"httpWorkerThread-4848-48" daemon prio=10 tid=0x00007efd18a82800 nid=0x5aa in Object.wait() [0x00007efd0db5a000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00007efd4d09a620> (a com.sun.enterprise.web.connector.grizzly.LinkedListPipeline)
    at java.lang.Object.wait(Object.java:485)
    at com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask(LinkedListPipeline.java:304)
    - locked <0x00007efd4d09a620> (a com.sun.enterprise.web.connector.grizzly.LinkedListPipeline)
    at com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run(WorkerThreadImpl.java:120)


"GrizzlyPipelineStatWorkerThread-4848-2" daemon prio=10 tid=0x00007efd20033000 nid=0x579 waiting on condition [0x00007efd10c8b000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00007efd4d098f30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
    at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
    at com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run(WorkerThreadImpl.java:113)

"GrizzlyPipelineStatWorkerThread-4848-1" daemon prio=10 tid=0x00007efd2124f000 nid=0x578 waiting on condition [0x00007efd10d8c000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00007efd4d098f30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
    at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
    at com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run(WorkerThreadImpl.java:113)

"httpSSLWorkerThread-8181-48" daemon prio=10 tid=0x0000000042041800 nid=0x575 in Object.wait() [0x00007efd1108f000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00007efd4d0c8308> (a com.sun.enterprise.web.connector.grizzly.ssl.SSLPipeline)
    at java.lang.Object.wait(Object.java:485)
    at com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask(LinkedListPipeline.java:304)
    - locked <0x00007efd4d0c8308> (a com.sun.enterprise.web.connector.grizzly.ssl.SSLPipeline)
    at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:109)


"SelectorThread-4848" prio=10 tid=0x00007efd207ba800 nid=0x546 runnable [0x00007efd13fbe000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    - locked <0x00007efd4d9c24c0> (a sun.nio.ch.Util$1)
    - locked <0x00007efd4d9c24a8> (a java.util.Collections$UnmodifiableSet)
    - locked <0x00007efd4d9c1d58> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    at com.sun.enterprise.web.connector.grizzly.SelectorThread.doSelect(SelectorThread.java:1421)
    at com.sun.enterprise.web.connector.grizzly.SelectorThread.startListener(SelectorThread.java:1336)
    - locked <0x00007efd4d9c27c0> (a [Ljava.lang.Object;)
    at com.sun.enterprise.web.connector.grizzly.SelectorThread.startEndpoint(SelectorThread.java:1299)
    at com.sun.enterprise.web.connector.grizzly.SelectorThread.run(SelectorThread.java:1275)


"httpSSLWorkerThread-8080-49" daemon prio=10 tid=0x00007efd20bfc800 nid=0x541 in Object.wait() [0x00007efd144c3000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00007efd4d0f97a8> (a com.sun.enterprise.web.portunif.PortUnificationPipeline)
    at java.lang.Object.wait(Object.java:485)
    at com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask(LinkedListPipeline.java:304)
    - locked <0x00007efd4d0f97a8> (a com.sun.enterprise.web.portunif.PortUnificationPipeline)
    at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:109)


"SelectorThread-8181" prio=10 tid=0x00007efd2161c000 nid=0x50e runnable [0x00007efd177f6000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    - locked <0x00007efd4d3a6800> (a sun.nio.ch.Util$1)
    - locked <0x00007efd4d3a67e8> (a java.util.Collections$UnmodifiableSet)
    - locked <0x00007efd4d3a6098> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    at com.sun.enterprise.web.connector.grizzly.SelectorThread.doSelect(SelectorThread.java:1421)
    at com.sun.enterprise.web.connector.grizzly.SelectorThread.startListener(SelectorThread.java:1336)
    - locked <0x00007efd4d3a6b00> (a [Ljava.lang.Object;)
    at com.sun.enterprise.web.connector.grizzly.SelectorThread.startEndpoint(SelectorThread.java:1299)
    at com.sun.enterprise.web.connector.grizzly.SelectorThread.run(SelectorThread.java:1275)

"GrizzlyPipelineStatWorkerThread-8080-1" daemon prio=10 tid=0x00007efd20f4c800 nid=0x50a waiting on condition [0x00007efd17bfa000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00007efd4d120680> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
    at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
    at com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run(WorkerThreadImpl.java:113)

"GrizzlyPipelineStatWorkerThread-8080-0" daemon prio=10 tid=0x00007efd20966800 nid=0x509 waiting on condition [0x00007efd17cfb000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00007efd4d120680> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
    at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
    at com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run(WorkerThreadImpl.java:113)

"SelectorThread-8080" prio=10 tid=0x00007efd201a0800 nid=0x508 runnable [0x00007efd17dfc000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    - locked <0x00007efd4dc827a0> (a sun.nio.ch.Util$1)
    - locked <0x00007efd4dc82788> (a java.util.Collections$UnmodifiableSet)
    - locked <0x00007efd4dc82088> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    at com.sun.enterprise.web.connector.grizzly.SelectorThread.doSelect(SelectorThread.java:1421)
    at com.sun.enterprise.web.connector.grizzly.SelectorThread.startListener(SelectorThread.java:1336)
    - locked <0x00007efd4dc82aa0> (a [Ljava.lang.Object;)
    at com.sun.enterprise.web.connector.grizzly.SelectorThread.startEndpoint(SelectorThread.java:1299)
    at com.sun.enterprise.web.connector.grizzly.SelectorThread.run(SelectorThread.java:1275)

"ContainerBackgroundProcessor[StandardEngine[com.sun.appserv]]" daemon prio=10 tid=0x00007efd207af800 nid=0x503 waiting on condition [0x00007efd1c76c000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1813)
    at java.lang.Thread.run(Thread.java:662)


"Timer-13" daemon prio=10 tid=0x00007efd202d2800 nid=0x4fe in Object.wait() [0x00007efd1c96e000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00007efd4ab21998> (a java.util.TaskQueue)
    at java.lang.Object.wait(Object.java:485)
    at java.util.TimerThread.mainLoop(Timer.java:483)
    - locked <0x00007efd4ab21998> (a java.util.TaskQueue)
    at java.util.TimerThread.run(Timer.java:462)

"Timer-9" daemon prio=10 tid=0x00007efd209ad000 nid=0x4f9 in Object.wait() [0x00007efd1d7d1000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00007efd4a82c750> (a java.util.TaskQueue)
    at java.util.TimerThread.mainLoop(Timer.java:509)
    - locked <0x00007efd4a82c750> (a java.util.TaskQueue)
    at java.util.TimerThread.run(Timer.java:462)

"Thread-11" daemon prio=10 tid=0x00007efd183f6800 nid=0x4c0 waiting on condition [0x00007efd1efe7000]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00007efd48273738> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    at com.sun.enterprise.management.support.LoaderRegThread.processFOREVER(LoaderRegThread.java:263)
    at com.sun.enterprise.management.support.LoaderRegThread.run(LoaderRegThread.java:311)


"SelectorThread" daemon prio=10 tid=0x00007efd183ff800 nid=0x4ba runnable [0x00007efd1f5ed000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    - locked <0x00007efd48171e70> (a sun.nio.ch.Util$1)
    - locked <0x00007efd48171e58> (a java.util.Collections$UnmodifiableSet)
    - locked <0x00007efd48171a98> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    at com.sun.corba.ee.impl.transport.SelectorImpl.run(SelectorImpl.java:283)

I'm looking for general information about interpreting the dump so that I can diagnose problems in the future. Thanks.

Allan Lykke Christensen about 12 years

Could you expand a bit on that? Do you have a reference to the bug?
danidacila about 12 years

issues.apache.org/bugzilla/show_bug.cgi?id=52858 a colleague submitted this issue with tomcat and they fixed it. There are similar for jetty and other containers. It is described as a JDK bug, but all containers implement some workaround to get the Selector out of spinning. If you search for Jetty and the epollWait I think you'll find the most explanations.