WinDbg x64: Cannot debug a crash dump - failed to load data access DLL

debugging windbg dump

38,589

Solution 1

I hit this regularly when debugging with minidumps from the site. Quite how it's happened in your case I'm not sure. Usually, it happens when the version of CLR that was loaded when the dump was taken is not available on your debugging machine. In your case, they're the same machine, so it should all probably just work. I'm sure there will be others who can explain exactly why it isn't.

In the meantime, here's what I do with my site dumps. Windbg is looking for "the right version" of mscordacwks.dll. So we give it that version and tell it where to look for it.

First - if I spoof all of this, by deleting mscordacwks.dll, windbg goes off and loads it from the Microsoft symbol server, so do make sure your symbols are set up correctly to download symbols from the Microsoft symbol server and give it another go.

Now - assuming that didn't work, check exactly which version is the "right version". List the module info with "lm v clr" and check your CLR version that is ACTUALLY loaded. Mine is 4.0.30319.239. Ok - now find that version of mscordacwks.dll. Let's assume it can be found in the normal .NET framework installation on your machine (C:\Windows\Microsoft.NET\Framework64\v4.0.30319). Do check the version matches exactly (right-click, properties etc)! Take it and put it in a safe place (I use D:\Symbols\_Images). Follow the instructions that windbg gave you on renaming the file. mscordacwks_.dll would be mscordacwks_AMD64_AMD64_4.0.30319.239.dll.

Now set up your executable image path (".exepath D:\Symbols\_Images") so windbg knows where you've put it.

You've now got "the right version of mscordacwks", and renamed it so that Windbg knows what it's looking for, and told it where you've put it.

If that STILL isn't working, then try ".cordll -ve -u -l" and also "!sym noisy" to turn on verbose logging of both the cordll load and the symbol server, then try !CLRStack again. Maybe the output of those two commands will tell you exactly what it's trying to load and you can figure out why it won't do it...

Solution 2

I just spent the day debugging a bunch of cases where we ran into this scenario. The SOS+CLR on the same box as the crash were unable to load within WinDbg, and "lm v" reported two different versions for the same module:

0:011> lm vM *clr.dll
start             end                 module name
000007fe`f2f50000 000007fe`f38b0000   clr      # (pdb symbols)          c:\symbols\clr.pdb\EDFF900AC9B94C1D9B32696A7759891A2\clr.pdb
    Loaded symbol image file: clr.dll
    Image path: C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll
    Image name: clr.dll
    Timestamp:        Sun Apr 21 03:36:04 2013 (5173C114)
    CheckSum:         0095F379
    ImageSize:        00960000
    File version:     4.0.30319.18052
    Product version:  4.0.30319.18052
    File flags:       8 (Mask 3F) Private
    File OS:          4 Unknown Win32
    File type:        2.0 Dll
    File date:        00000000.00000000
    Translations:     0409.04b0
    CompanyName:      Microsoft Corporation
    ProductName:      Microsoft® .NET Framework
    InternalName:     clr.dll
    OriginalFilename: clr.dll
    ProductVersion:   4.0.30319.18047
    FileVersion:      4.0.30319.18047 built by: FX45RTMGDR
    PrivateBuild:     DDBLD320
    FileDescription:  Microsoft .NET Runtime Common Language Runtime - WorkStation
    LegalCopyright:   © Microsoft Corporation.  All rights reserved.
    Comments:         Flavor=Retail

Backing Details

The file Timestamp (0x5173C114), Checksum (0x0095F379), and Version (4.0.30319.18052) stored in the MINIDUMP_MODULE structure in the minidump's module-list-stream was for the newer CLR. Cracking open the minidump file myself and looking directly at the stream data:

MINIDUMP_MODULE : (pack:8 size:112) 
  +0x000 .BaseOfImage UInt64 : 8791579230208 (0x7FEF2F50000)
  +0x008 .SizeOfImage UInt32 : 9830400 (0x960000)
  +0x00C .CheckSum UInt32 : 9827193 (0x95F379)
  +0x010 .TimeDateStamp UInt32 : 1366540564 (0x5173C114)
  +0x014 .ModuleNameRva UInt32 : 107828 (0x1A534)
  +0x018 .VersionInfo tagVS_FIXEDFILEINFO : (pack:8 size:52) 
    +0x000 .dwSignature UInt32 : 4277077181 (0xFEEF04BD)
    +0x004 .dwStrucVersion UInt32 : 65536 (0x10000)
    +0x008 .dwFileVersionMS UInt32 : 262144 (0x40000)
    +0x00C .dwFileVersionLS UInt32 : 1987004036 (0x766F4684)

Splitting the high and low words out of dwFileVersionMS we get 4 and 0.
Splitting the high and low words out of dwFileVersionLS we get 30319 and 18052.

Using dumpchk.exe, and looking at the module details in the PEB, we can see a different Timestamp (0x515530CE), one that actually corresponds to the older (18047) version:

7fef2f50000 515530ce Mar 28 23:12:30 2013 C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll

Looking at the raw memory in the crash dump where clr.dll is loaded, you can see the Checksum (0x00965F80) and Timestamp (0x515530CE) of version 4.0.30319.18047:

0:011> db 000007fe`f2f50000 
000007fe`f2f50000  4d 5a 90 00 03 00 00 00-04 00 00 00 ff ff 00 00  MZ..............
000007fe`f2f50010  b8 00 00 00 00 00 00 00-40 00 00 00 00 00 00 00  ........@.......
000007fe`f2f50020  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000007fe`f2f50030  00 00 00 00 00 00 00 00-00 00 00 00 18 01 00 00  ................
000007fe`f2f50040  0e 1f ba 0e 00 b4 09 cd-21 b8 01 4c cd 21 54 68  ........!..L.!Th
000007fe`f2f50050  69 73 20 70 72 6f 67 72-61 6d 20 63 61 6e 6e 6f  is program canno
000007fe`f2f50060  74 20 62 65 20 72 75 6e-20 69 6e 20 44 4f 53 20  t be run in DOS 
000007fe`f2f50070  6d 6f 64 65 2e 0d 0d 0a-24 00 00 00 00 00 00 00  mode....$.......
0:011> db
000007fe`f2f50080  39 e4 28 ed 7d 85 46 be-7d 85 46 be 7d 85 46 be  9.(.}.F.}.F.}.F.
000007fe`f2f50090  81 f2 f8 be 79 85 46 be-81 f2 fa be 74 85 46 be  ....y.F.....t.F.
000007fe`f2f500a0  74 fd c5 be 73 85 46 be-74 fd c2 be c9 85 46 be  t...s.F.t.....F.
000007fe`f2f500b0  ee 41 8d be 7f 85 46 be-e3 25 81 be 7c 85 46 be  .A....F..%..|.F.
000007fe`f2f500c0  ee 41 88 be 6b 85 46 be-ee 41 89 be 78 85 46 be  .A..k.F..A..x.F.
000007fe`f2f500d0  ee 41 8b be 64 85 46 be-7d 85 47 be ca 87 46 be  .A..d.F.}.G...F.
000007fe`f2f500e0  81 f2 ff be 76 85 46 be-ee 41 9e be 70 87 46 be  ....v.F..A..p.F.
000007fe`f2f500f0  ee 41 8c be 7c 85 46 be-ee 41 8f be 7c 85 46 be  .A..|.F..A..|.F.
0:011> 
000007fe`f2f50100  ee 41 8a be 7c 85 46 be-52 69 63 68 7d 85 46 be  .A..|.F.Rich}.F.
000007fe`f2f50110  00 00 00 00 00 00 00 00-50 45 00 00 64 86 06 00  ........PE..d...
000007fe`f2f50120  ce 30 55 51 00 00 00 00-00 00 00 00 f0 00 22 20  .0UQ.........." 
000007fe`f2f50130  0b 02 0b 00 00 90 69 00-00 c2 2b 00 00 00 00 00  ......i...+.....
000007fe`f2f50140  40 51 13 00 00 10 00 00-00 00 f5 f2 fe 07 00 00  @Q..............
000007fe`f2f50150  00 10 00 00 00 02 00 00-06 00 00 00 0a 00 00 00  ................
000007fe`f2f50160  06 00 00 00 00 00 00 00-00 e0 95 00 00 04 00 00  ................
000007fe`f2f50170  80 5f 96 00 02 00 60 01-00 00 10 00 00 00 00 00  ._....`.........

I also jumped ahead in memory and looked at the Version resource in memory and saw the 18047 version string.

So now we have a minidump with conflicting information about what version of clr.dll was actually in use.

What Caused It

I also found out that our IT department recently pushed out a handful of Windows Updates, so:

While an application was running, an update to the CLR was installed.
The file in C:\Windows\Microsoft.NET\Framework64\v4.0.30319\ was updated to the newer version (4.0.30319.18052)
The application Crashed
MiniDumpWriteDump apparently used the file-on-disk information when storing the module list in the crash dump file (4.0.30319.18052)
WinDbg wasn't able to correlate the version stamped in the crash dump with what was in the process' memory as it had conflicting information.

To verify this, I manually modified the MINIDUMP_MODULE entry for clr.dll to change the Checksum, Timestamp, and Version from 18052 to 18047. After reloading the hacked up .dmp file in WinDbg and setting the exepath to the proper sos+clr dlls, I was able to successfully execute the sos commands and get a valid stack trace.

Bottom Line

We essentially ended up with a minidump file that has conflicting information about which version of clr.dll is loaded into the process, preventing SOS from identifying the correct clr engine. This is likely a bug in MiniDumpWriteDump as it saves out the crash dump file. But should only happen when the backing version of the CLR has been updated while your application was running.

38,589

Author by

net_prog

.NET/C# programmer.

Updated on July 09, 2022

Comments

net_prog almost 2 years

I attached WinDbg to a running process and had the process crashed (I have a separate question re. that case). Once the program crashed, WinDbg stopped and allowed me to debug the program. I took a crash dump for further investigation with a command ".dump /ma".

The program was compiled as "Any CPU" and I used WinDbg x64 to take the dump. Now I open WinDbg x64 on the same computer again and open the crash dump. Here is what it says:

Loading Dump File [C:\crashdump.dmp]
User Mini Dump File with Full Memory: Only application data is available

Symbol search path is: SRV*c:\symbols*http://msdl.microsoft.com/download/symbols
Executable search path is: 
Windows 7 Version 7601 (Service Pack 1) MP (8 procs) Free x64
Product: WinNt, suite: SingleUserTS
Machine Name:
Debug session time: Mon Aug 15 10:24:57.000 2011 (UTC + 1:00)
System Uptime: 17 days 0:54:39.021
Process Uptime: 12 days 14:01:31.000
................................................................
...............................................................
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(1be0.b78): Access violation - code c0000005 (first/second chance not available)
*** WARNING: symbols timestamp is wrong 0x4dd2333e 0x4da4281c for clr.dll
clr!WKS::gc_heap::find_first_object+0x92:
000007fe`ea129a1d f70100000080    test    dword ptr [rcx],80000000h ds:00000000`00003d80=????????

Then I try to load SOS by ".load sos clr" and it errors in:

The call to LoadLibrary(sos clr) failed, Win32 error 0n2
    "The system cannot find the file specified."
Please check your debugger configuration and/or network access.

Trying with ".loadby sos clr" and it works. Now I want to see the stack with "!clrstack" and stick here:

Failed to load data access DLL, 0x80004005
Verify that 1) you have a recent build of the debugger (6.2.14 or newer)
            2) the file mscordacwks.dll that matches your version of clr.dll is 
                in the version directory
            3) or, if you are debugging a dump file, verify that the file 
                mscordacwks_<arch>_<arch>_<version>.dll is on your symbol path.
            4) you are debugging on the same architecture as the dump file.
                For example, an IA64 dump file must be debugged on an IA64
                machine.

You can also run the debugger command .cordll to control the debugger's
load of mscordacwks.dll.  .cordll -ve -u -l will do a verbose reload.
If that succeeds, the SOS command should work on retry.

If you are debugging a minidump, you need to make sure that your executable
path is pointing to clr.dll as well.

I tried ".symfix" and ".reload":

0:027> .symfix
0:027> .reload
..................*** WARNING: symbols timestamp is wrong 0x4dd2333e 0x4da4281c for clr.dll
..............................................
...............................................................

Stuck. At the same time when the process is running under WinDgb I can pause the execution, load SOS and execute "!clrstack" command successfully.

Any ideas? Thank you.

UPDATE - Followed the steps provided in the second answer, still doesn't work.

1) My Symbol Path: SRV*c:\symbols*http://msdl.microsoft.com/download/symbols;srv*

2) CLR loaded: 4.0.30319.237:

0:027> lm v clr
Unknown option 'r'
start             end                 module name
00000000`77b60000 00000000`77d09000   ntdll      (pdb symbols)          c:\symbols\ntdll.pdb\6192BFDB9F04442995FFCB0BE95172E12\ntdll.pdb
    Loaded symbol image file: ntdll.dll
    Image path: C:\Windows\System32\ntdll.dll
    Image name: ntdll.dll
    Timestamp:        Sat Nov 20 13:11:21 2010 (4CE7C8F9)
    CheckSum:         001B55EA
    ImageSize:        001A9000
    File version:     6.1.7601.17514
    Product version:  6.1.7601.17514
    File flags:       0 (Mask 3F)
    File OS:          40004 NT Win32
    File type:        2.0 Dll
    File date:        00000000.00000000
    Translations:     0409.04b0
    CompanyName:      Microsoft Corporation
    ProductName:      Microsoft® Windows® Operating System
    InternalName:     ntdll.dll
    OriginalFilename: ntdll.dll
    ProductVersion:   6.1.7601.17514
    FileVersion:      6.1.7601.17514 (win7sp1_rtm.101119-1850)
    FileDescription:  NT Layer DLL
    LegalCopyright:   © Microsoft Corporation. All rights reserved.
000007fe`e9fb0000 000007fe`ea915000   clr      # (pdb symbols)          c:\symbols\clr.pdb\1A7EA01DA29549DAB2B0BD012A6C5BA12\clr.pdb
    Loaded symbol image file: clr.dll
    Image path: C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll
    Image name: clr.dll
    Timestamp:        Tue May 17 09:35:10 2011 (4DD2333E)
    CheckSum:         00967144
    ImageSize:        00965000
    File version:     4.0.30319.237
    Product version:  4.0.30319.237
    File flags:       8 (Mask 3F) Private
    File OS:          4 Unknown Win32
    File type:        2.0 Dll
    File date:        00000000.00000000
    Translations:     0409.04b0
    CompanyName:      Microsoft Corporation
    ProductName:      Microsoft® .NET Framework
    InternalName:     clr.dll
    OriginalFilename: clr.dll
    ProductVersion:   4.0.30319.235
    FileVersion:      4.0.30319.235 (RTMGDR.030319-2300)
    PrivateBuild:     DDBLD240
    FileDescription:  Microsoft .NET Runtime Common Language Runtime - WorkStation
    LegalCopyright:   © Microsoft Corporation.  All rights reserved.
    Comments:         Flavor=Retail

3) "C:\Windows\Microsoft.NET\Framework64\v4.0.30319\mscordacwks.dll" has version 4.0.30319.239

4) I found that when I load the dump into WinDbg it loads the correct "mscordacwks.dll" from the web, thus in the folder "C:\symbols\mscordacwks_AMD64_AMD64_4.0.30319.237.dll\4DD2333E965000" I have the file "mscordacwks_AMD64_AMD64_4.0.30319.237.dll".

0:027> .cordll -ve -u -l
CLR DLL status: No load attempts

0:027> !sym noisy
noisy mode - symbol prompts on
0:027> .restart

Loading Dump File [C:\crashdump.dmp]
User Mini Dump File with Full Memory: Only application data is available

DBGHELP: Symbol Search Path: srv*;srv*c:\symbols*http://msdl.microsoft.com/download/symbols
DBGHELP: Symbol Search Path: cache*;SRV*http://msdl.microsoft.com/download/symbols;srv*c:\symbols*http://msdl.microsoft.com/download/symbols
DBGHELP: Symbol Search Path: cache*;SRV*http://msdl.microsoft.com/download/symbols;srv*c:\symbols*http://msdl.microsoft.com/download/symbols
Symbol search path is: srv*;SRV*c:\symbols*http://msdl.microsoft.com/download/symbols
Executable search path is: 
Windows 7 Version 7601 (Service Pack 1) MP (8 procs) Free x64
Product: WinNt, suite: SingleUserTS
Machine Name:
Debug session time: Mon Aug 15 10:24:57.000 2011 (UTC + 1:00)
System Uptime: 17 days 0:54:39.021
Process Uptime: 12 days 14:01:31.000
................................................................
...............................................................
DBGHELP: ntdll - public symbols  
         C:\Program Files\Debugging Tools for Windows (x64)\sym\ntdll.pdb\6192BFDB9F04442995FFCB0BE95172E12\ntdll.pdb
DBGHELP: Symbol Search Path: cache*;SRV*http://msdl.microsoft.com/download/symbols;srv*c:\symbols*http://msdl.microsoft.com/download/symbols
DBGHELP: Symbol Search Path: cache*;SRV*http://msdl.microsoft.com/download/symbols;srv*c:\symbols*http://msdl.microsoft.com/download/symbols
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(1be0.b78): Access violation - code c0000005 (first/second chance not available)
*** WARNING: symbols timestamp is wrong 0x4dd2333e 0x4da4281c for clr.dll
DBGHELP: clr - public symbols  
         C:\Program Files\Debugging Tools for Windows (x64)\sym\clr.pdb\1A7EA01DA29549DAB2B0BD012A6C5BA12\clr.pdb
clr!WKS::gc_heap::find_first_object+0x92:
000007fe`ea129a1d f70100000080    test    dword ptr [rcx],80000000h ds:00000000`00003d80=????????

0:027> !clrstack
SYMSRV:  C:\Program Files\Debugging Tools for Windows (x64)\sym\mscordacwks_AMD64_AMD64_4.0.30319.237.dll\4DD2333E965000\mscordacwks_AMD64_AMD64_4.0.30319.237.dll not found
SYMSRV:  mscordacwks_AMD64_AMD64_4.0.30319.237.dll from http://msdl.microsoft.com/download/symbols: 502892 bytes - copied         
DBGHELP: C:\Program Files\Debugging Tools for Windows (x64)\sym\mscordacwks_AMD64_AMD64_4.0.30319.237.dll\4DD2333E965000\mscordacwks_AMD64_AMD64_4.0.30319.237.dll cached to C:\Program Files\Debugging Tools for Windows (x64)\sym\mscordacwks_AMD64_AMD64_4.0.30319.237.dll\4DD233F317b000\mscordacwks_AMD64_AMD64_4.0.30319.237.dll
DBGHELP: C:\Program Files\Debugging Tools for Windows (x64)\sym\mscordacwks_AMD64_AMD64_4.0.30319.237.dll\4DD233F317b000\mscordacwks_AMD64_AMD64_4.0.30319.237.dll - OK
Failed to load data access DLL, 0x80004005
Verify that 1) you have a recent build of the debugger (6.2.14 or newer)
            2) the file mscordacwks.dll that matches your version of clr.dll is 
                in the version directory
            3) or, if you are debugging a dump file, verify that the file 
                mscordacwks_<arch>_<arch>_<version>.dll is on your symbol path.
            4) you are debugging on the same architecture as the dump file.
                For example, an IA64 dump file must be debugged on an IA64
                machine.

You can also run the debugger command .cordll to control the debugger's
load of mscordacwks.dll.  .cordll -ve -u -l will do a verbose reload.
If that succeeds, the SOS command should work on retry.

If you are debugging a minidump, you need to make sure that your executable
path is pointing to clr.dll as well.

net_prog over 12 years

I just checked and all the components are installed. Also when I attach WinDbg to a running process and then have the program crash everything works fine which also means all the components are there I think.
net_prog over 12 years

Followed the steps you described, still can't make it work, please see the updated question.
Pete over 12 years

No immediate thoughts, except to wonder why the CLR loaded (presumably from the normal .NET installation) is 237, and yet the mscordacwks.dll loaded (presumably from the same place) is 239. I'll give it a ponder. I thought of this page: Doug Stewart's version history of CLR
Pete over 12 years

The debugger is saying that there is a mismatch of the timestamp in the symbol file for clr.dll (and yet you are using microsoft's symbol server). Also, you've version 237 of clr.dll locally, and 239 of mscordacwks.dll locally (even though the debugger's getting the 'right' version ok). I wonder if reinstalling the update for .NET 4.0.20319.239 (as linked in Doug's blog post) might just make everything work as expected. Unless you're using a "special" patch from microsoft for another reason of course.
Pete over 12 years

Or maybe you've even been hit by malware - your clr.dll doesn't match microsoft's symbols and it really should, so all bets are off. A quick scan with some respectable anti-malware can't hurt!
net_prog over 12 years

The process was compiled for Any CPU and it a 64 bit process.
Pete over 12 years

Doug Stewart just added this post, which also links to this post, which also has a load of other links to try out - you might give them a quick glance.
Thomas Weller almost 11 years

It is either .loadby <plugin> <module> or .load <plugin_full_path>, but never .load sos clr.