C# Service locks-up (Memory/Resource Management/GC Problem?)

[Keep it to Usenet please]

I've inherited a service written in C# that will eventually become
non-responsive (Has to be "Force Quit" -- can't be stopped/restarted
from the Services Control Panel.). It really seems like what would be
a problem with Memory/Resource Management in C++ -- memory not being
freed, files not being closed, etc.. If this was C/C++, I'd use one
of the tried and trued malloc wrappers, custom
constructors/destructors, etc..

Other than a source code crawl to look for things that need to be
explicitly .Dispose()ed, is there a better way to track something like
this down?

 

[Andreas Huber]

I've inherited a service written in C# that will eventually become
non-responsive (Has to be "Force Quit" -- can't be stopped/restarted
from the Services Control Panel.).

The non-responsiveness can have a multitude of reasons, failure to
release resources being only one of them. If you're sure that it's that,
it would be interesting to hear the symptoms (e.g. process memory,
handle count, etc. keeps going up).

It really seems like what would be
a problem with Memory/Resource Management in C++ -- memory not being
freed, files not being closed, etc.. If this was C/C++, I'd use one
of the tried and trued malloc wrappers, custom
constructors/destructors, etc..

Other than a source code crawl to look for things that need to be
explicitly .Dispose()ed, is there a better way to track something like
this down?

Not disposing objects of classes that implement IDisposable only means
that the resource release process becomes non-deterministic. As long as
root references no longer point to the object directly or indirectly,
the GC *will* call the finalizers, which then release all resources
(assuming that the finalizers are correctly implemented). So, not
calling Dispose() is usually not the problem behind the failure to
release resources. More often it's a failure to set root references to
null (e.g. static collections or events). Root reference management
problems can be found with a memory profiler.

HTH,

 

[William Stacey [MVP]]

Have you ruled out an actual deadlock (Monitor) issue. May be able to see
where the hang is with Managed Stack Explorer:
http://go.microsoft.com/fwlink/?LinkId=59380

--
William Stacey [MVP]

| I've inherited a service written in C# that will eventually become
| non-responsive (Has to be "Force Quit" -- can't be stopped/restarted
| from the Services Control Panel.). It really seems like what would be
| a problem with Memory/Resource Management in C++ -- memory not being
| freed, files not being closed, etc.. If this was C/C++, I'd use one
| of the tried and trued malloc wrappers, custom
| constructors/destructors, etc..
|
| Other than a source code crawl to look for things that need to be
| explicitly .Dispose()ed, is there a better way to track something like
| this down?
|
| --
| A: No. See:
Help, I'm
| <http://www.netmeister.org/news/learn2quote.html> being
held
| <http://www.greenend.org.uk/rjk/2000/06/14/quoting> in a
..sig
| Q: Should I include quotations after my reply?
factory!

 

[greg.merideth]

While not a killer tool (for this use) I use the sys internals process
explorer to watch thread creation from within my c# services. Using PE
you can kill off individual threads from within your service to help
pin down issues.

 

[Keep it to Usenet please]

Have you ruled out an actual deadlock (Monitor) issue. May be able to see
where the hang is with Managed Stack Explorer:
http://go.microsoft.com/fwlink/?LinkId=59380

Unfortunately, this app is still .NET 1.1. Looking into migrating to
2.0 is one of the other tasks on my list, but they want the hanging
tracked down first.

 

[Keep it to Usenet please]

The non-responsiveness can have a multitude of reasons, failure to
release resources being only one of them. If you're sure that it's
that, it would be interesting to hear the symptoms (e.g. process
memory, handle count, etc. keeps going up).

I've just started on this task, and so far, I've not encountered a
lock-up yet. I've just been warned that it will happen. The only
symptoms I know of is that the service will stop responding, cannot be
stopped using the Services control panel and needs to be hunted down
and killed through the Task Manager.

 

[Andreas Huber]

I've just started on this task, and so far, I've not encountered a
lock-up yet. I've just been warned that it will happen. The only
symptoms I know of is that the service will stop responding, cannot be
stopped using the Services control panel and needs to be hunted down
and killed through the Task Manager.

I assume that the rest of the computer keeps working normally when that
happens, i.e. no memory shortage, etc. If so, then this looks more like
a dead-lock problem than a resource management problem. To get a hint
why the dead-lock happens, you could attach a debugger to the
non-responsive service (never tried that with services but it should
work in principle) and hit break. If you then examine all the
call-stacks of all threads you should find the dead-lock pretty quickly.

HTH & Regards,

 

[William Stacey [MVP]]

1) Maybe you could attach to it using VS debugger and use a debug build on
the service.

2) Managed Debug Assistants may be another tool.

3) If it is using an Monitors (i.e. lock()), then find them all and replace
any "forever" wait locks with timeouts, then throw exceptions on timeout so
you can find the offending code.

4) Have not tested it, but does just throwing an exception in a service
produce the same effect? Maybe need some more try/catch logic and logging.

5) Maybe Windbg and SOS:
http://msdn.microsoft.com/msdnmag/issues/03/06/Bugslayer/

6) Compile with 2.0. I understand you want to fix it in 1.1 first, but with
the updates and fixes in 2.0, you may find the issue with a simple compile
error and/or find the issue faster because your using the 2.0 runtime and
you might see different behavior that help you nail the issue. Use it as
more of a debug tool, then as a comittment to move to 2.0. Now you can also
use Managed Stack Explorer as a bonus! You got to do something, so this
would be my first pick as it helps elliminate a lot of variables up front
with minor effort. Also, during debugging, leave as a Console app instead
of a service - just easier to debug and test. Use existing stress tests to
see if you can get it to fail in lab. If not, develop some more tests.

--
William Stacey [MVP]

| In article <[email protected]>,
|
| > Have you ruled out an actual deadlock (Monitor) issue. May be able to
see
| > where the hang is with Managed Stack Explorer:
| > http://go.microsoft.com/fwlink/?LinkId=59380
|
| Unfortunately, this app is still .NET 1.1. Looking into migrating to
| 2.0 is one of the other tasks on my list, but they want the hanging
| tracked down first.
|
| --
| A: No. See:
Help, I'm
| <http://www.netmeister.org/news/learn2quote.html> being
held
| <http://www.greenend.org.uk/rjk/2000/06/14/quoting> in a
..sig
| Q: Should I include quotations after my reply?
factory!

 

[Keep it to Usenet please]

6) Compile with 2.0. ...

It may be already. It looked like 2.0 wasn't installed, but looking
at "Add/Remove Programs", .NET 2.0 has been installed. I'm not sure
of the best way to determine if the code has been migrated to 2.0 or
not.

 

[Keep it to Usenet please]

I assume that the rest of the computer keeps working normally when
that happens, i.e. no memory shortage, etc.

AFAIK, correct.

If so, then this looks more like a dead-lock problem than a
resource management problem. To get a hint why the dead-lock
happens, you could attach a debugger to the non-responsive service
(never tried that with services but it should work in principle)
and hit break. If you then examine all the call-stacks of all
threads you should find the dead-lock pretty quickly.

Hopefully, they'll be patient enough when this happens. So far,
they've not been able to duplicate the problem in test. The
assumption is that they're not able to generate the same volume in
test as is seen in live, and that's why they never see the freezes on
the test server.