Windbg, how to read the !locks output?

Question

I am debugging a program which I suspect there could be deadlock or other mutli-thread related bug, I follow people's suggestions to use WinDBG to open the crash dump file and used !locks to get the following output:

CritSec MSVCR100D!lclcritsects+48 at 73541e40
WaiterWoken        No
LockCount          6
RecursionCount     1
OwningThread       164c
EntryCount         0
ContentionCount    9
*** Locked

*** ERROR: Symbol file could not be found.  Defaulted to export symbols for qsqlited4.dll - 
CritSec qsqlited4!qt_plugin_instance+a1b21 at 70fc301c
WaiterWoken        No
LockCount          0
RecursionCount     1
OwningThread       2344
EntryCount         0
ContentionCount    0
*** Locked

CritSec +73c2380 at 073c2380
WaiterWoken        No
LockCount          0
RecursionCount     4
OwningThread       2344
EntryCount         0
ContentionCount    0
*** Locked

CritSec +73bf9e8 at 073bf9e8
WaiterWoken        No
LockCount          0
RecursionCount     1
OwningThread       2344
EntryCount         0
ContentionCount    0    
*** Locked

Scanned 817 critical sections

I am confused by the output, could anyone help to explain it?

Solution 1

!locks can be confusing. If you really want to debug a deadlock situation, do a ~*kvn (or kb whichever you like) find threads waiting on critical sections which will end in a **WaitForSingleForSingleObject and before that a RtlEnterCriticalSection call. Find the Critical section most of the threads are wating on. Dump the critical section. If you are debugging x64 based dumps and narrow down to the frame which is carrying RtlCrticalSection using .frame /c post you are in thread context ~[threadnum]s, rbx will contain your critical section.

Dump the critical section find the owner. If the owner is waiting find out what's owner waiting on and so on till we reach end of chain or a reason why things are blocked. !cs -l -o can be confusing if we don't put it in context.

Hope this helps.

Solution 2

Teb is address to the thread environment block, Suspend & frozen not relevant for now

Assuming that it’s a 32 bit scenario, you can reveal which critical section a thread is waiting by:

a) Switch to the thread
b) dump stack
c) Find 1 argument to RtlEnterCriticalSection

(If 64 follow the recept from Addy above)

Solution 3

To find deadlocks caused by critical sections, try SOSEX's !dlk command. Although the extension seems to be for .NET only, the !dlk command will also identify deadlocks in native critical sections.

Benefit: if it identifies a deadlock, it's very easy to read. If it does not, you still need to apply other techniques (e.g. in case the chain includes other types of synchronization objects).

Example output (not specifically for critical section):

0:010> !dlk
Deadlock detected:
CLR thread 4 holds sync block 00000000024c6970 OBJ:000000007fff0f80[System.String] STRVAL=SYNC1
             waits sync block 00000000024c6928 OBJ:000000007fff0fa8[System.String] STRVAL=SYNC2
CLR thread 5 holds sync block 00000000024c6928 OBJ:000000007fff0fa8[System.String] STRVAL=SYNC2
             waits sync block 00000000024c6970 OBJ:000000007fff0f80[System.String] STRVAL=SYNC1
CLR Thread 4 is waiting at ConsoleTestApp.ConsoleTestApp.MonitorDeadlockThreadProc()+0xa4(IL) [C:\dev\ConsoleTestApp\ConsoleTestApp.cs, line 195]
CLR Thread 5 is waiting at ConsoleTestApp.ConsoleTestApp.MonitorDeadlockThreadProc()+0xa4(IL) [C:\dev\ConsoleTestApp\ConsoleTestApp.cs, line 195]

1 deadlock detected.