Low CPU hang in GC

[scott]

My mixed-mode .net 2.0 service is hanging during a GC and I need some help
trying to find out why. My service has 6 managed threads. Thread 1 is the
service control thread, thread 6 is in the middle of doing a GC and is
blocking, and all other managed threads are trying to do a GC but are waiting
on thread 6 to finish. Effectively all threads are hung. After waiting for
2 minutes and taking another dump, the threads haven't moved. I need to find
out what thread 6 is blocking on. windbg shows the following stack:

0:005> kb
ChildEBP RetAddr Args to Child
0397ea14 7d4d8c9e 000031dc 00000000 00000000 ntdll!ZwWaitForSingleObject+0x15
0397ea84 79e8c639 000031dc ffffffff 00000000
kernel32!WaitForSingleObjectEx+0xac
0397eac8 79e8c56f 000031dc ffffffff 00000000 mscorwks!PEImage::LoadImage+0x1af
0397eb18 79e8c58e ffffffff 00000000 00000000 mscorwks!CLREvent::WaitEx+0x117
0397eb2c 79f2f90f ffffffff 00000000 00000000 mscorwks!CLREvent::Wait+0x17
0397eb4c 79f2f94a 7a3b9020 00000000 ffffffff
mscorwks!WKS::gc_heap::user_thread_wait+0x50
0397eb5c 79f2f886 00000000 7a3b8b18 00080101
mscorwks!WKS::gc_heap::wait_to_proceed+0xe
0397eb70 79f92fea 00000000 00000000 00000000
mscorwks!WKS::gc_heap::garbage_collect+0x247
0397eb9c 79f94eb6 00000000 00000000 00000020
mscorwks!WKS::GCHeap::GarbageCollectGeneration+0x1a9
0397ec28 79f9275e 0033faa0 00000020 00000000
mscorwks!WKS::gc_heap::try_allocate_more_space+0x15b
0397ec3c 79f927f9 0033faa0 00000020 00000000
mscorwks!WKS::gc_heap::allocate_more_space+0x11
0397ec5c 79e732c1 0033faa0 00000020 00000002 mscorwks!WKS::GCHeap::Alloc+0x3b
0397ec78 79e7c5d2 00000020 00000000 00080000 mscorwks!Alloc+0x60
0397ecb8 79e7c697 7932b01c 5e4475c5 0d9d295c mscorwks!FastAllocateObject+0x38
0397ed5c 792f1c61 0d9d29fc 0d9d279c 01f6641c mscorwks!JIT_NewFast+0x9e
0397ed74 7a9e029a 00000000 00000000 0d9d2928 mscorlib_ni+0x231c61
0397eda8 7a9e6204 00000000 00000000 00000000 System_ni+0x5a029a
0397eddc 7a9e5f01 01f6641c 01f66440 00000000 System_ni+0x5a6204
0397ee18 7a9e558f 00000000 01f66580 0d9d25a4 System_ni+0x5a5f01
0397ee38 792e5eb1 01f66580 0397ee4c 7a9e555b System_ni+0x5a558f

I can see that it's waiting on handle 000031dc, but windbg doesn't tell me
much about what that handle is:

0:005> !handle 000031dc f
Handle 000031dc
Type Event
Attributes 0
GrantedAccess 0x1f0003:
Delete,ReadControl,WriteDac,WriteOwner,Synch
QueryState,ModifyState
HandleCount 2
PointerCount 4
Name <none>
No object specific information available

Any help would be appreciated. Thanks!

 

[Jason Evans]

Hello Scott,

Don't know for sure if this will make a difference, but I spotted the use
of the !handle command here: http://blogs.msdn.com/ntdebugging/archive/2008/03/18/how-to-find-the-owner-of-a-named-pipe.aspx
and it was used like this

!handle ObjectAddress f ProcessAddress

Maybe adding the process address at the end of the command might give you
more information?

Cheers.
Jas.

 

[scott]

Hi Jason,

Thanks for the feedback, but I believe the process address parameter is for
kernel debugging only and these are user mode dumps. That may be why more
information isn't available from the handles in these dumps...

I've opened up a support call with Microsoft on this case. From what they
can tell, it looks like the thread doing the actual garbage collecting has
exited without signaling the waiting thread that triggered the GC. This
causes that thread to hang forever. The current theory is that something
caused the GC thread to exit abnormally... possibly because of memory
corruption. According to MS, our best chance of troubleshooting this is to
take a hang dump of the GC thread right before it exists. Since it's tricky
to reproduce, it may be a while before I find out the real cause.

 

[Scott M.]

Are you manually calling the GC?

 

[scott]

Are you manually calling the GC?

You mean, GC.Collect()? Nope.

 

[Scott M.]

Yes, that's what I meant. Ok, good.

You mean, GC.Collect()? Nope.