Programmatic deadlock detection in java

Question

How can I programmatically detect that a deadlock has occurred in a Java program?

Solution 1

You can do this programmatically using the ThreadMXBean that ships with the JDK:

ThreadMXBean bean = ManagementFactory.getThreadMXBean();
long[] threadIds = bean.findDeadlockedThreads(); // Returns null if no threads are deadlocked.

if (threadIds != null) {
    ThreadInfo[] infos = bean.getThreadInfo(threadIds);

    for (ThreadInfo info : infos) {
        StackTraceElement[] stack = info.getStackTrace();
        // Log or store stack trace information.
    }
}

Obviously you should try to isolate whichever thread is performing this deadlock check - Otherwise if that thread deadlocks it won't be able to run the check!

Incidentally this is what JConsole is using under the covers.

Solution 2

One useful hint for investigation:

If you can catch the application red handed and suspect a deadlock has occurred, go and press "Ctrl-Break" in the java.exe console window (or "Ctrl-\" on Solaris/Linux). The jvm will dump the current status and stack trace of all threads, find out dead locks and precisely describe them.

It will look something like this:

Full thread dump Java HotSpot(TM) Client VM (1.5.0_09-b03 mixed mode):

"[Test Timer] Request Queue" prio=6 tid=0x13d708d0 nid=0x1ec in Object.
    wait() [0x1b00f000..0x1b00fb68]
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Unknown Source)
    at library.util.AsyncQueue.run(AsyncQueue.java:138)
        - locked <0x02e70000> (a test.server.scheduler.SchedulerRequestQueue)

    ...

Found one Java-level deadlock:
=============================
"Corba service":
  waiting to lock monitor 0x13c06684 (object 0x04697d90, a java.lang.Object),
  which is held by "[Server Connection] Heartbeat Timer"
"[Server Connection] Heartbeat Timer":
  waiting to lock monitor 0x13c065c4 (object 0x0467e728, a test.proxy.ServerProxy), which is held by "Corba service"

Java stack information for the threads listed above:
===================================================
"Corba service":
    at test.proxy.ServerProxy.stopHBWatchDog(ServerProxy:695)
    - waiting to lock <0x04697d90> (a java.lang.Object)
    ...

Solution 3

You can detect the deadlocked threads programmatically using ThreadMXBean class.Here is the code,

    ThreadMXBean bean = ManagementFactory.getThreadMXBean();

    long ids[] = bean.findMonitorDeadlockedThreads();

    if(ids != null)
    {
        ThreadInfo threadInfo[] = bean.getThreadInfo(ids);

        for (ThreadInfo threadInfo1 : threadInfo)
        {
            System.out.println(threadInfo1.getThreadId());    //Prints the ID of deadlocked thread

            System.out.println(threadInfo1.getThreadName());  //Prints the name of deadlocked thread

            System.out.println(threadInfo1.getLockName());    //Prints the string representation of an object for which thread has entered into deadlock.

            System.out.println(threadInfo1.getLockOwnerId());  //Prints the ID of thread which currently owns the object lock

            System.out.println(threadInfo1.getLockOwnerName());  //Prints name of the thread which currently owns the object lock.
        }
    }
    else
    {
        System.out.println("No Deadlocked Threads");
    }

Click here for more info on how to detect the deadlocked threads.

Solution 4

JArmus is a library for deadlock detection and avoidance. It includes support for: Thread.join, CyclicBarrier, CountDownLatch, Phaser, and ReentrantLock.

To use JArmus you need to instrument your code. Either through one of its instrumented classes or automatically with the JArmus instrumentar jarmusc.

java -jar jarmusc.jar yourprogram.jar checkedprogram.jar

The input yourprogram.jar is the program you want to check. The output is the same program with checks to automatically find any deadlock.

Barriers need some help

Verifying deadlocks with classes CyclicBarrier, CountDownLatch, Phaser is a bit tricky --- for example, JConsole cannot detect these types of deadlocks. JArmus needs a little help from you: you must specify which threads are influencing synchronization, we call these registered threads.

As soon as possible, the thread must mark itself as registered. A good place to mark registered threads is at the beginning method Runnable.run. JArmus.register(latch);

Example

The following program that deadlocks is correctly identified by JArmus:

final CountDownLatch latch = new CountDownLatch(2);
final CyclicBarrier barrier = new CyclicBarrier(2);
final Queue<Exception> exceptions = new ArrayDeque<>();
Thread t1 = new Thread(new Runnable() {
    @Override
    public void run() {
        try {
            JArmus.register(barrier); // do not forget to register!
            JArmus.register(latch); // do not forget to register!
            latch.countDown();
            latch.await();
            barrier.await();
        } catch (Exception e) {
            exceptions.add(e);
        }
    }
});
Thread t2 = new Thread(new Runnable() {
    @Override
    public void run() {
        try {
            JArmus.register(barrier); // do not forget to register!
            JArmus.register(latch); // do not forget to register!
            barrier.await();
            latch.countDown();
            latch.await();
        } catch (Exception e) {
            exceptions.add(e);
        }
    }
});
t1.start();
t2.start();

Solution 5

If you don't require programmatic detection you can do this via the JConsole; on the thread tab there is a "detect deadlock" button. In JDK6 this detect locks for both intrinsic monitors and j.u.c Locks

Run up the JConsole via the $JAVA_HOM/bin/jconsole command

View more solutions

Comments

[Brett]

Or ctrl-\ from Solaris/Linux.

[SyntaxT3rr0r]

Actually what you want is send a "QUIT" signal (ie a SIGQUIT) but it's not really a 'QUIT'. A SIGQUIT on OS X, Linux (and Solaris) on a Java app dumps the stacktrace. Ctrl+\ is one way to send a SIGQUIT. Taking the pid and doing: kill -3 {id} is another way to do it.

[Bart van Heukelom]

The Javadoc for that method says it might be expensive. Do you happen to know about how expensive?

[Adamski]

No I don't although I guess you could write some code to time the operation and see how this scales as thread count / number of locks increases.

[korhner]

There are a few problems with this code. First, findDeadlockedThreads could return null for some threads which can cause NPE in your loop. Second, this will not print actual stack trace, you need to get actual threads from thread info. You can check this post for more info: korhner.github.io/java/multithreading/…

[Adamski]

@korhner: findDeadlockedThreads() will return null in the case of there being no deadlocked threads, which I already check for; I don't believe it ever returns an array containing sporadic null values. getThreadInfo can return an array containing nulls if the corresponding threads are not alive. However, I already know they are alive (and deadlocked) from the previous call. Re. the second point I've left stack trace logging as an exercise for the OP but they don't actually mention that they want to do this.

[korhner]

@Adamski:findDeadlockedThreads can report deadlocks even if a thread has a timeout on a lock which in practice is not a real dedlock (after timeout lock is released). In theory it can happen that the thread is destroyed after timeout and before your getThreadInfo. You are right about findDeadlockedThreads won't return null for specific threads however, sorry.

[Amit Kumar Gupta]

It is working for me in some cases. But not for all cases.

[Stefan Reich]

The "MTRAT" link goes to a page about floating-point arithmetic. Confused what that's about

[Stefan Reich]

I benchmarked the findDeadlockedThreads method (on a HotSpot Server VM with only few threads, admittedly). Each call takes less than 100 µs after warmup. tinybrain.de/1009294

[Stefan Reich]

Benchmark results for more threads: For 100 sleeping threads the detection takes <250 µs. For 1000 sleeping threads it takes <4 ms. Still very OK to do occasionally.

[Henrik Steudel]

In case you wondered why stack trace is empty in above example: Use getThreadInfo(threadIds, Integer.MAX_VALUE) to acquire all stack trace elements.

[Investigator]

Better to use findDeadlockedThreads() instead of the findMonitorDeadlockedThreads(). The findDeadlockedThreads has been introduced on 1.6 and is better in comparison with the findMonitorDeadlockedThreads() that may loose some deadlock cases and not report them. It is not able to report deadlocks for: ReentrankLock or ReentrantReadWriteLock. More info can be found here: meteatamel.wordpress.com/2012/03/21/deadlock-detection-in-ja‌​va under the "findMonitorDeadlockedThreads vs. findDeadlockedThreads" title