Random crashes in mscorwks.dll...

[Guest]

I really am banging my head with this one...

I have a regular unmanaged C++ application that is using mixed DLLs to
(amongst other things) call a C# back-end (which is using ADO .NET). I am
getting random crashes, typically after 15-60 minutes of inactivity. The
error message and call stack are listed below.

Unhandled exception at 0x792483e0 (mscorwks.dll) in
debug_build_wks_crash.dmp: 0xC0000005: Access violation reading location
0xfeef00da.

mscorwks.dll!CorExitProcess() + 0x5e2c
mscorwks.dll!ND_CopyObjDst() + 0x2fa9
mscorwks.dll!CorExitProcess() + 0x4a3c
mscoree.dll!__CorDllMain@12() + 0x1c
ntdll.dll!_LdrpCallInitRoutine@16() + 0x14
ntdll.dll!_LdrShutdownThread@0() + 0xdb
kernel32.dll!_ExitThread@4() + 0x3e
mscorwks.dll!ReleaseFusionInterfaces() + 0x44a7f
mscorwks.dll!ReleaseFusionInterfaces() + 0x441b0
kernel32.dll!_BaseThreadStart@8() + 0x37

In an attempt to fix or identify the problem I have:

1. Searched the code for places where unmanaged code could be corrupting
managed memory.
2. Tracked (using Trace.WriteLine()) garbage collection. This seemed to
indicate that garbage collection (of my objects) had taken place long before
the crash.
3. Implemented the workaround for the mixed DLL loading problem
(http://support.microsoft.com/?id=814472).

I can't make much sense of the call stack, even this
(http://msdn.microsoft.com/library/d...y/en-us/cpgenref/html/grfuncorexitprocess.asp) document doesn't really help me.

I would really appreciate either suggestions as to what might be causing the
problem or ideas that might help me track it down.

Thanks

 

[Oleg Starodumov]

It looks like there is no symbols for mscorwks.dll on your system
(exports are loaded instead), and therefore the call stack is incorrect.
You should obtain the good symbols (e.g. download them from
the symbol server) to get the correct call stack.

Regards,
Oleg

 

[Guest]

Sorry... The call stack was created from a dump file which for some reason
didn't pick up the symbols when I opened it in VS. Here is the actual call
stack (I have typed it from a screen-shot and triple checked for typos):

mscorwks.dll!Thread::UnhijackThread() + 0x70d9f
mscorwks.dll!Thread::RareEnablePreemptiveGC() + 0x36
mscorwks.dll!Thread::RareDisablePreemptiveGC() + 0x49cb6
mscorwks.dll!SystemDomain::RunDllMain() + 0x7d
mscorwks.dll!ExecuteDLL() + 0x7fd12
mscoree.dll!__CorDllMain@12() + 0x1c
ntdll.dll!_LdrpCallInitRoutine@16() + 0x14
ntdll.dll!_LdrShutdownThread@0() + 0xdb
kernel32.dll!_ExitThread@4() + 0x3e
mscorwks.dll!ThreadpoolMgr::WorkerThreadStart() + 0x123
mscorwks.dll!ThreadpoolMgr::intermediateThreadProc() + 0x44
kernel32.dll!_BaseThreadStart@8() + 0x37

I have since tried using pageheap
(http://support.microsoft.com/default.aspx?scid=kb;en-us;286470 and
http://support.microsoft.com/default.aspx?scid=kb;en-us;264471) to detect
memory corruption in the application. Despite identifying and fixing a memory
problem the application still crashes in the same way.

I will be very gratefull for any help you can provide...

 

[Vladimir Nesterovsky]

Hello,

I believe some problem araises during thread (from thread pool)
finalization. May be this is somehow related to an unmanaged code that was
executed in that thread.
Can you replace (for testing purposes) asynchronous calls to explicit
threads. The earlier you will finalize these threads, the better.

 

[Oleg Starodumov]

In addition, try to run these commands in Command Window
after you have opened the dump:

..load sos
!DumpStack

What call stack will be shown?

Regards,
Oleg

 

[Guest]

The ouput is:

..load sos

!DumpStack
succeeded
Loaded Son of Strike data table version 5 from
"C:\WINDOWS\Microsoft.NET\Framework\v1.1.4322\mscorwks.dll"
Current frame: ?ReturnToContext@Thread@@QAEXPAVFrame@@H@Z+0x2d
ChildEBP RetAddr Caller,Callee
0428f6d8 792a5ea4 ?RunDllMain@SystemDomain@@SGJPAUHINSTANCE__@@KPAX@Z+0x288,
calling ?ReturnToContext@Thread@@QAEXPAVFrame@@H@Z
0428f78c 79246ff0 ?ExecuteDLL@@YGHPAUHINSTANCE__@@KPAX@Z+0x3c0, calling
?RunDllMain@SystemDomain@@SGJPAUHINSTANCE__@@KPAX@Z
0428f7b0 792438ac _DllMain@12+0x33, calling
?EEDllMain@@YGHPAUHINSTANCE__@@KPAX@Z
0428f7c4 791c049c _DllMain@12+0x117, calling @__security_check_cookie@4
0428f7c8 79b91ba8 (stub for mdToken: 0600002a (), calling 79b7a188
0428f7cc 79a2bbe7 79a2bbe7, calling 79b91ba3 (MethodDesc 0x79b91ba8
mdToken: 0600002a ()
0428f7dc 791b1b3a ?JIT_MonExit@@YIXPAVObject@@@Z+0xf, calling
?DummyGetThread@@YGPAVThread@@XZ
0428f81c 791ed194
?CallDescr@MethodDesc@@QAE_JPBEPAVModule@@PAVMetaSig@@HPB_J@Z+0x1b8, calling
_CallDescrWorker@16
0428f850 791daddf ??0ArgIterator@@QAE@PAEPAVMetaSig@@H@Z+0x55, calling
?IsArgumentInRegister@@YGHPAHEIHE0@Z
0428f864 791d94bc _CallDescrWorker@16+0x30
0428f86c 791ed194
?CallDescr@MethodDesc@@QAE_JPBEPAVModule@@PAVMetaSig@@HPB_J@Z+0x1b8, calling
_CallDescrWorker@16
0428f874 791ed1cd
?CallDescr@MethodDesc@@QAE_JPBEPAVModule@@PAVMetaSig@@HPB_J@Z+0x1f1, calling
?getFPReturn@@YGXHAA_J@Z
0428f890 77f944cb _RtlpFreeToHeapLookaside@8+0x1f, calling
@RtlpInterlockedPushEntrySList@8
0428f898 77f58bcd _RtlFreeHeap@12+0x18f, calling _RtlpFreeToHeapLookaside@8
0428f8a0 77f59037 _RtlFreeHeap@12+0x5f9, calling __SEH_epilog
0428f8c0 77f944cb _RtlpFreeToHeapLookaside@8+0x1f, calling
@RtlpInterlockedPushEntrySList@8
0428f8c8 77f58bcd _RtlFreeHeap@12+0x18f, calling _RtlpFreeToHeapLookaside@8
0428f8d0 77f59037 _RtlFreeHeap@12+0x5f9, calling __SEH_epilog
0428f8e0 77f59037 _RtlFreeHeap@12+0x5f9, calling __SEH_epilog
0428f8ec 79513737 _DllMain@12+0x119, calling @__security_check_cookie@4
0428f910 77f944cb _RtlpFreeToHeapLookaside@8+0x1f, calling
@RtlpInterlockedPushEntrySList@8
0428f918 77f58bcd _RtlFreeHeap@12+0x18f, calling _RtlpFreeToHeapLookaside@8
0428f920 77f59037 _RtlFreeHeap@12+0x5f9, calling __SEH_epilog
0428f938 77f59037 _RtlFreeHeap@12+0x5f9, calling __SEH_epilog
0428f960 77f59037 _RtlFreeHeap@12+0x5f9, calling __SEH_epilog
0428fa14 7917cff3 __CorDllMain@12+0x40
0428fa2c 77f5b42c _LdrpCallInitRoutine@16+0x14
0428fa4c 77f629df _LdrShutdownThread@0+0xdb, calling _LdrpCallInitRoutine@16
0428fa74 7929b73f
?SetDelayedInheritedSecurityStack@Thread@@QAEXPAVCompressedStack@@@Z+0x1f,
calling ?Release@CompressedStack@@QAEJXZ
0428fac8 77e74af8 _ExitThread@4+0x3e, calling _LdrShutdownThread@0
0428fb20 792ec273 ?intermediateThreadProc@ThreadpoolMgr@@CGKPAX@Z+0x44
0428fb2c 771256e2 _DllMain@12+0x27, calling
?_typesDllMain@@YGHPAUHINSTANCE__@@KPAX@Z
0428fb74 77f58a3a _RtlAllocateHeap@12+0xe8c, calling __SEH_epilog
0428fb88 762fa952 ?I_CryptTlsDllMain@@YGHPAUHINSTANCE__@@KPAX@Z+0x19f,
calling __SEH_epilog
0428fbac 76317375 ?CryptPFXDllMain@@YGHPAUHINSTANCE__@@KPAX@Z+0x24, calling
?EncodeDecodeDllMain@@YGHPAUHINSTANCE__@@KPAX@Z
0428fbd8 77f58a3a _RtlAllocateHeap@12+0xe8c, calling __SEH_epilog
0428fbdc 76f611cd _ldapMalloc@8+0x17, calling _LdrpDllTagProcedures+0x1
0428fbf0 76f61234 _AddPerThreadEntry@4+0x4d, calling
_RtlComputeImportTableHash@12+0x18
0428fc00 76f610e6 _LdapDllInit@12+0x67, calling _AddPerThreadEntry@4
0428fc08 77f5b42c _LdrpCallInitRoutine@16+0x14
0428fc30 77f62b67 _LdrpInitializeThread@4+0xe3, calling
_RtlDeactivateActivationContextUnsafeFast@4
0428fc38 77f62bc6 _LdrpInitializeThread@4+0x142, calling __SEH_epilog
0428fc9c 77f55432 _LdrpInitialize@12+0x26f, calling _RtlLeaveCriticalSection@4
0428fca0 77f5c474 _ZwTestAlert@0+0xc
0428fca4 77f5541d _LdrpInitialize@12+0x25a, calling __SEH_epilog
0428fd18 77f5541d _LdrpInitialize@12+0x25a, calling __SEH_epilog
0428fd1c 77f5b644 _ZwContinue@8+0xc
0428fd20 77f75d8f _KiUserApcDispatcher@20+0xf, calling _ZwContinue@8
0428fdd4 77f944a8 _RtlpAllocateFromHeapLookaside@4+0x42, calling __SEH_epilog
0428fdd8 77f57d70 _RtlAllocateHeap@12+0x1c2, calling
_RtlpAllocateFromHeapLookaside@4
0428ffa4 792ec261 ?intermediateThreadProc@ThreadpoolMgr@@CGKPAX@Z+0x32,
calling __alloca_probe
0428ffb4 77e7d28e _BaseThreadStart@8+0x37

I have managed to "solve" the problem by doing a complete "rebuild" i.e.
clearing my development folder and getting files from scratch from source
control. I believe that the problem may be related to the fact that I had 23
different versions of a managed DLL in various locations with five different
timestamps. VS copies managed DLLs when it is building and I don't think the
copies were being correctly updated.

I have accepted your answer because you have provided me with an additional
mechanism to try and identify the problem. I can't make much sense of the
managed call stack but if there are any hints in there please let me know.

Thanks for taking the time to post...

 

[Guest]

Update - we are still working to resolve this. The issue seems to be related
to (i) the use of connection pooling and (ii) running the application on dual
processor machines.

 

[Shy]

*Update - we are still working to resolve this. The issue seems to be
related
to (i) the use of connection pooling and (ii) running the application
on dual
processor machines. *

I'm having a similar problem. In my case, whenever it happens I stopped
the application and tried to find the renegade thread in the visual
studio.
It appears that it's not even one of mine...
Does anyone know what can be done?

 

[Guest]

My apologies for not updating this when we finally found a "solution". With
the help of MCS we solved the problem by disabling thread attach and detach
notification for the troublesome DLLs.

This was some time ago and I have since changed jobs so cannot provide much
more information than that. I do remember that Microsoft were very helpful in
getting to the bottom of the problem.

P.S. I hope you don't *need* thread attach and detach notifications... ;-)