What would be the cause of this worker process crash?
14,159
Solution 1
The clue to this problem is in the Exception Code:
Sig[6].Name=Exception Code Sig[6].Value=e053534f <====== this is the clue
This exception code indicates that there was a "soft stack overflow" raised by the CLR. Please see the commentary at the end of this MSDN Library article:
What you'll need to do is install the Debug Diagnostic Toolkit and configure it to dump your worker process whenever this crash happens.
Once you've captured a dump of the process you'll need to load this into WinDBG and use SOS to narrow down the root cause.
There are several great articles on how to analyse a worker process crash dump:
.NET Debugging Demos - Information and setup instructions - Tess Ferrandez
Diagnosing Tricky ASP.Net Production Issues with DebugDiag and winDbg
System.StackOverflowException with IronPython 2.6.0 and 2.6.1
The articles by Tess Ferrandez are great and I learned to master WinDBG and SOS from her labs.
Solution 2
To add to Kev's excellent answer -
Right Click > My Computer > Properties > Advanced Tab > click the Settings button under Start up and Recovery > choose "Complete Memory Dump" and specify a location with enough HDD space equal to the amount of RAM in the server.
Once the crash has reoccurred and the memory dump is successful. Install WinDBG, open up WinDBG, File > Open > load the memory dump (.dmp)
Then load SOS (eg .loadby sos mscorwks)
!clrstack
This will churn out a huge stack trace that will most likely identify a recursive function or some huge allocation as Brian Grunkemeyer - MSFT said in the link Kev provided.
Your probably going to have to put a limit on the depth of the recursive calls... or something similar.
Good luck! You can always call Microsoft PSS if you need help with the diagnosis. Remember to download the Public Symbols from Microsoft before you issue !analyze and !clrstack - in order to see method names. If you need to see method bodies in Microsoft DLLs in a dump file you need the private symbols (only avail to Microsoft engineers).
Author by
Admin
Updated on June 29, 2022Comments
-
Admin almost 2 years
We experienced a worker process crash in our ASP.NET 2.0 application.
Here is the report that was dumped when the worker process crashed:
Version=1 EventType=APPCRASH EventTime=129575741278397484 ReportType=2 Consent=1 ReportIdentifier=7c124dac-c466-11e0-a346-68b599c966f4 WOW64=1 Response.type=4 Sig[0].Name=Application Name Sig[0].Value=w3wp.exe Sig[1].Name=Application Version Sig[1].Value=7.5.7600.16385 Sig[2].Name=Application Timestamp Sig[2].Value=4a5bcd2b Sig[3].Name=Fault Module Name Sig[3].Value=KERNELBASE.dll Sig[4].Name=Fault Module Version Sig[4].Value=6.1.7600.16823 Sig[5].Name=Fault Module Timestamp Sig[5].Value=4de7264e Sig[6].Name=Exception Code Sig[6].Value=e053534f Sig[7].Name=Exception Offset Sig[7].Value=0000b9bc DynamicSig[1].Name=OS Version DynamicSig[1].Value=6.1.7600.2.0.0.272.7 DynamicSig[2].Name=Locale ID DynamicSig[2].Value=1033 UI[2]=C:\Windows\SysWOW64\inetsrv\w3wp.exe LoadedModule[0]=C:\Windows\SysWOW64\inetsrv\w3wp.exe LoadedModule[1]=C:\Windows\SysWOW64\ntdll.dll LoadedModule[2]=C:\Windows\syswow64\kernel32.dll LoadedModule[3]=C:\Windows\syswow64\KERNELBASE.dll LoadedModule[4]=C:\Windows\syswow64\ADVAPI32.dll LoadedModule[5]=C:\Windows\syswow64\msvcrt.dll LoadedModule[6]=C:\Windows\SysWOW64\sechost.dll LoadedModule[7]=C:\Windows\syswow64\RPCRT4.dll LoadedModule[8]=C:\Windows\syswow64\SspiCli.dll LoadedModule[9]=C:\Windows\syswow64\CRYPTBASE.dll LoadedModule[10]=C:\Windows\system32\pcwum.DLL LoadedModule[11]=C:\Windows\syswow64\USER32.dll LoadedModule[12]=C:\Windows\syswow64\GDI32.dll LoadedModule[13]=C:\Windows\syswow64\LPK.dll LoadedModule[14]=C:\Windows\syswow64\USP10.dll LoadedModule[15]=C:\Windows\syswow64\ole32.dll LoadedModule[16]=C:\Windows\SysWOW64\inetsrv\IISUTIL.dll LoadedModule[17]=C:\Windows\system32\IMM32.DLL LoadedModule[18]=C:\Windows\syswow64\MSCTF.dll LoadedModule[19]=C:\Windows\system32\ntmarta.dll LoadedModule[20]=C:\Windows\syswow64\WLDAP32.dll LoadedModule[21]=C:\Windows\SysWOW64\inetsrv\w3wphost.dll LoadedModule[22]=C:\Windows\syswow64\OLEAUT32.dll LoadedModule[23]=C:\Windows\SysWOW64\inetsrv\nativerd.dll LoadedModule[24]=C:\Windows\syswow64\CRYPT32.dll LoadedModule[25]=C:\Windows\syswow64\MSASN1.dll LoadedModule[26]=C:\Windows\system32\XmlLite.dll LoadedModule[27]=C:\Windows\system32\ktmw32.dll LoadedModule[28]=C:\Windows\SysWOW64\inetsrv\IISRES.DLL LoadedModule[29]=C:\Windows\system32\CRYPTSP.dll LoadedModule[30]=C:\Windows\system32\rsaenh.dll LoadedModule[31]=C:\Windows\system32\mscoree.dll LoadedModule[32]=C:\Windows\syswow64\CLBCatQ.DLL LoadedModule[33]=C:\Windows\system32\mlang.dll LoadedModule[34]=C:\Windows\syswow64\SHLWAPI.dll LoadedModule[35]=C:\Windows\Microsoft.NET\Framework\v2.0.50727\mscorwks.dll LoadedModule[36]=C:\Windows\WinSxS\x86_microsoft.vc80.crt_1fc8b3b9a1e18e3b_8.0.50727.4927_none_d08a205e442db5b5\MSVCR80.dll LoadedModule[37]=C:\Windows\system32\inetsrv\iiscore.dll LoadedModule[38]=C:\Windows\SysWOW64\inetsrv\W3TP.dll LoadedModule[39]=C:\Windows\SysWOW64\inetsrv\w3dt.dll LoadedModule[40]=C:\Windows\system32\HTTPAPI.dll LoadedModule[41]=C:\Windows\system32\slc.dll LoadedModule[42]=C:\Windows\syswow64\WS2_32.dll LoadedModule[43]=C:\Windows\syswow64\NSI.dll LoadedModule[44]=C:\Windows\syswow64\Normaliz.dll LoadedModule[45]=C:\Windows\system32\faultrep.dll LoadedModule[46]=C:\Windows\system32\Secur32.dll LoadedModule[47]=C:\Windows\system32\NLAapi.dll LoadedModule[48]=C:\Windows\System32\mswsock.dll LoadedModule[49]=C:\Windows\system32\DNSAPI.dll LoadedModule[50]=C:\Windows\System32\winrnr.dll LoadedModule[51]=C:\Windows\system32\napinsp.dll LoadedModule[52]=C:\Windows\System32\wshtcpip.dll LoadedModule[53]=C:\Windows\System32\wship6.dll LoadedModule[54]=C:\Windows\system32\IPHLPAPI.DLL LoadedModule[55]=C:\Windows\system32\WINNSI.DLL LoadedModule[56]=C:\Windows\system32\rasadhlp.dll LoadedModule[57]=C:\Windows\System32\fwpuclnt.dll LoadedModule[58]=C:\Windows\System32\inetsrv\cachuri.dll LoadedModule[59]=C:\Windows\System32\inetsrv\cachfile.dll LoadedModule[60]=C:\Windows\System32\inetsrv\cachtokn.dll LoadedModule[61]=C:\Windows\System32\inetsrv\cachhttp.dll LoadedModule[62]=C:\Windows\System32\inetsrv\compstat.dll LoadedModule[63]=C:\Windows\System32\inetsrv\defdoc.dll LoadedModule[64]=C:\Windows\System32\inetsrv\dirlist.dll LoadedModule[65]=C:\Windows\System32\inetsrv\protsup.dll LoadedModule[66]=C:\Windows\System32\inetsrv\static.dll LoadedModule[67]=C:\Windows\System32\inetsrv\authanon.dll LoadedModule[68]=C:\Windows\System32\inetsrv\modrqflt.dll LoadedModule[69]=C:\Windows\System32\inetsrv\custerr.dll LoadedModule[70]=C:\Windows\System32\inetsrv\loghttp.dll LoadedModule[71]=C:\Windows\System32\inetsrv\iisreqs.dll LoadedModule[72]=C:\Windows\system32\WSOCK32.dll LoadedModule[73]=C:\Windows\System32\inetsrv\isapi.dll LoadedModule[74]=C:\Windows\System32\inetsrv\filter.dll LoadedModule[75]=C:\Windows\System32\inetsrv\validcfg.dll LoadedModule[76]=C:\Windows\Microsoft.NET\Framework\v2.0.50727\webengine.dll LoadedModule[77]=C:\Windows\system32\USERENV.dll LoadedModule[78]=C:\Windows\system32\profapi.dll LoadedModule[79]=C:\Windows\syswow64\PSAPI.DLL LoadedModule[80]=C:\Windows\System32\inetsrv\iisetw.dll LoadedModule[81]=C:\Windows\System32\inetsrv\iisfreb.dll LoadedModule[82]=C:\Windows\System32\inetsrv\compdyn.dll LoadedModule[83]=C:\Windows\System32\inetsrv\redirect.dll LoadedModule[84]=C:\Windows\System32\inetsrv\iis_ssi.dll LoadedModule[85]=C:\Windows\System32\inetsrv\authcert.dll LoadedModule[86]=C:\Windows\System32\inetsrv\urlauthz.dll LoadedModule[87]=C:\Windows\System32\inetsrv\authbas.dll LoadedModule[88]=C:\Windows\System32\inetsrv\authsspi.dll LoadedModule[89]=C:\Windows\system32\NETAPI32.dll LoadedModule[90]=C:\Windows\system32\netutils.dll LoadedModule[91]=C:\Windows\system32\srvcli.dll LoadedModule[92]=C:\Windows\system32\wkscli.dll LoadedModule[93]=C:\Windows\System32\inetsrv\authmd5.dll LoadedModule[94]=C:\Windows\System32\inetsrv\authmap.dll LoadedModule[95]=C:\Windows\System32\inetsrv\iprestr.dll LoadedModule[96]=C:\Windows\System32\inetsrv\logcust.dll LoadedModule[97]=C:\Windows\Microsoft.NET\Framework\v2.0.50727\aspnet_filter.dll LoadedModule[98]=C:\Windows\system32\inetsrv\wbhst_pm.dll LoadedModule[99]=C:\Windows\syswow64\shell32.dll LoadedModule[100]=C:\Windows\assembly\NativeImages_v2.0.50727_32\mscorlib\23bc3936180ff789f44259a211dfc7fc\mscorlib.ni.dll LoadedModule[101]=C:\Windows\assembly\NativeImages_v2.0.50727_32\System\bb2b3b9c7c2941a3f485940d59a68131\System.ni.dll LoadedModule[102]=C:\Windows\assembly\NativeImages_v2.0.50727_32\System.Web\d2d39614c7251ce46674ecc4516e9e85\System.Web.ni.dll LoadedModule[103]=C:\Windows\system32\sxs.dll LoadedModule[104]=C:\Windows\system32\RpcRtRemote.dll LoadedModule[105]=C:\Windows\assembly\NativeImages_v2.0.50727_32\System.Configuration\4c57c8b1594281c44d9f04a3b3d76131\System.Configuration.ni.dll LoadedModule[106]=C:\Windows\assembly\NativeImages_v2.0.50727_32\System.Xml\447fd8040376beceb8f2572214434843\System.Xml.ni.dll LoadedModule[107]=C:\Windows\system32\version.dll LoadedModule[108]=C:\Windows\assembly\NativeImages_v2.0.50727_32\Microsoft.JScript\c1d036e7d21aa16c6ecd5903f1380245\Microsoft.JScript.ni.dll LoadedModule[109]=C:\Windows\assembly\NativeImages_v2.0.50727_32\System.ServiceModel\fe0eb1e786dd2cb1af265642f98987df\System.ServiceModel.ni.dll LoadedModule[110]=C:\Windows\assembly\NativeImages_v2.0.50727_32\System.Core\800db9c1d9899d341145e2610d0fdec9\System.Core.ni.dll LoadedModule[111]=C:\Windows\assembly\NativeImages_v2.0.50727_32\System.Web.Abstract#\6c76cacd8bb7624ae8cc37da12b20c31\System.Web.Abstractions.ni.dll LoadedModule[112]=C:\Windows\assembly\NativeImages_v2.0.50727_32\System.Web.Extensio#\9e14cbd9cedcd676a5846b7bb3f00578\System.Web.Extensions.ni.dll LoadedModule[113]=C:\Windows\assembly\GAC_32\Oracle.Web\2.112.1.2__89b483f429c47342\Oracle.Web.dll LoadedModule[114]=C:\Windows\assembly\NativeImages_v2.0.50727_32\SMDiagnostics\b2de5b6831274468b42d17ab83649810\SMDiagnostics.ni.dll LoadedModule[115]=C:\Windows\system32\inetsrv\gzip.dll LoadedModule[116]=C:\Windows\assembly\NativeImages_v2.0.50727_32\System.Web.Services\c19548b56ec8119d2f41e057548a8e19\System.Web.Services.ni.dll LoadedModule[117]=C:\Windows\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\webservices_echoengine\4740617c\23f17d74\assembly\dl3\1ef7f3cf\398c7100_ed58ca01\SonarEngine.DLL LoadedModule[118]=C:\Windows\assembly\NativeImages_v2.0.50727_32\System.Data\179b93bc916180c4648a44b6fd79a1de\System.Data.ni.dll LoadedModule[119]=C:\Windows\assembly\GAC_32\System.Data\2.0.0.0__b77a5c561934e089\System.Data.dll LoadedModule[120]=C:\Windows\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\webservices_echoengine\4740617c\23f17d74\assembly\dl3\cf1595f1\a896187d_e657ca01\clsEchoEngine.DLL LoadedModule[121]=C:\Windows\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\webservices_echoengine\4740617c\23f17d74\assembly\dl3\48523473\64b3c57e_e657ca01\EchoEnginePool.DLL LoadedModule[122]=C:\Windows\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\webservices_echoengine\4740617c\23f17d74\assembly\dl3\022778b1\aaa284e5_992aca01\EpsiXml.DLL LoadedModule[123]=C:\Windows\assembly\GAC_32\Oracle.DataAccess\2.112.1.2__89b483f429c47342\Oracle.DataAccess.dll LoadedModule[124]=C:\Windows\assembly\NativeImages_v2.0.50727_32\System.Drawing\1c9d526783dc991a4c0db0adde215f77\System.Drawing.ni.dll LoadedModule[125]=C:\Windows\assembly\NativeImages_v2.0.50727_32\System.EnterpriseSe#\57d7078bc796f91103e92b210de3c3d6\System.EnterpriseServices.ni.dll LoadedModule[126]=C:\Windows\system32\apphelp.dll LoadedModule[127]=C:\Windows\Microsoft.NET\Framework\v2.0.50727\mscorjit.dll LoadedModule[128]=C:\Windows\assembly\NativeImages_v2.0.50727_32\System.Data.SqlXml\e121c1e73fabfce6f229546388585865\System.Data.SqlXml.ni.dll LoadedModule[129]=C:\Windows\assembly\NativeImages_v2.0.50727_32\Microsoft.VisualBas#\4e3e6b682ceb418593e5964993baf846\Microsoft.VisualBasic.ni.dll LoadedModule[130]=C:\Windows\assembly\NativeImages_v2.0.50727_32\System.Transactions\33eef8d1899573df9ab38acc0ffff07a\System.Transactions.ni.dll LoadedModule[131]=C:\Windows\assembly\GAC_32\System.Transactions\2.0.0.0__b77a5c561934e089\System.Transactions.dll LoadedModule[132]=D:\app\inetAdmin\product\11.2.0\client_1\OCI.DLL LoadedModule[133]=D:\app\inetAdmin\product\11.2.0\client_1\bin\OraOps11w.dll LoadedModule[134]=D:\app\inetAdmin\product\11.2.0\client_1\OraOCIEI11.dll LoadedModule[135]=C:\Windows\system32\WINMM.dll LoadedModule[136]=C:\Windows\system32\dbghelp.dll LoadedModule[137]=C:\Windows\assembly\NativeImages_v2.0.50727_32\Microsoft.Vsa\b933f490039e14eb2e4be2e597ae07ed\Microsoft.Vsa.ni.dll LoadedModule[138]=C:\Windows\Microsoft.NET\Framework\v2.0.50727\mscorpe.dll LoadedModule[139]=C:\Windows\system32\cscapi.dll LoadedModule[140]=C:\Windows\assembly\NativeImages_v2.0.50727_32\System.EnterpriseSe#\57d7078bc796f91103e92b210de3c3d6\System.EnterpriseServices.Wrapper.dll LoadedModule[141]=C:\Windows\assembly\GAC_32\System.EnterpriseServices\2.0.0.0__b03f5f7f11d50a3a\System.EnterpriseServices.Wrapper.dll LoadedModule[142]=C:\Windows\system32\Com\comadmin.dll LoadedModule[143]=C:\Windows\system32\MfcSubs.dll LoadedModule[144]=C:\Windows\WinSxS\x86_microsoft.windows.gdiplus_6595b64144ccf1df_1.1.7600.16385_none_72fc7cbf861225ca\gdiplus.dll LoadedModule[145]=C:\Windows\system32\WindowsCodecs.dll LoadedModule[146]=C:\Windows\system32\windowscodecsext.dll LoadedModule[147]=C:\Windows\System32\msxml6.dll LoadedModule[148]=C:\Windows\system32\mscms.dll LoadedModule[149]=C:\Windows\system32\icm32.dll LoadedModule[150]=C:\Windows\Microsoft.NET\Framework\v2.0.50727\diasymreader.dll LoadedModule[151]=C:\Windows\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\webservices_echoengine\4740617c\23f17d74\assembly\dl3\295de4c1\4c9fc2e5_992aca01\EchoMQ.DLL LoadedModule[152]=C:\Windows\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\webservices_echoengine\4740617c\23f17d74\assembly\dl3\6bdf2035\b19152e5_992aca01\RegistryAccessDLL.DLL LoadedModule[153]=C:\Windows\assembly\NativeImages_v2.0.50727_32\System.Messaging\59069ccdbc65fce202bb976999326efa\System.Messaging.ni.dll LoadedModule[154]=C:\Windows\system32\mqrt.dll LoadedModule[155]=C:\Windows\system32\mqsec.dll LoadedModule[156]=C:\Windows\system32\LOGONCLI.DLL LoadedModule[157]=C:\Windows\system32\SAMCLI.DLL LoadedModule[158]=C:\Windows\system32\DSROLE.DLL LoadedModule[159]=C:\Windows\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\webservices_echoengine\4740617c\23f17d74\App_Web_defaultwsdlhelpgenerator.aspx.cdcab7d2.cvuegqau.dll LoadedModule[160]=C:\Windows\assembly\NativeImages_v2.0.50727_32\System.Web.Mobile\7967455a68fba1ebdccdcc88a49c7f11\System.Web.Mobile.ni.dll LoadedModule[161]=C:\Windows\assembly\NativeImages_v2.0.50727_32\System.Web.RegularE#\32f0e82ec89a541cd35b1199026bdc79\System.Web.RegularExpressions.ni.dll FriendlyEventName=APPCRASH ConsentKey=APPCRASH AppName=IIS Worker Process AppPath=C:\Windows\SysWOW64\inetsrv\w3wp.exe ReportDescription=Stopped workingThis occurred on our server running Windows 2008 R2 64 bit.
What caused this error and how do I prevent it from happening?
-
Thomas Weller over 9 yearsStep 1 applies only to kernel dumps, not user mode dumps. Instead set up WER LocalDumps