Service terminating silently

[Jon Skeet [C# MVP]]

Thought I'd throw this out and see if anyone's seen anything similar.

I have a Windows service written in C#, which occasionally stops
silently. The process goes away, Windows notices and shows it as
unstarted in the service control manager.

This always happens (as far as we've seen) after another specific kind
of error (an external resource being unavailable), but the time between
the previous error and the service failing can be hours. The previous
error is logged nicely, but there's nothing in the log just before the
service dies.

I've tried reproducing this by writing deliberately bad code, and seen
the following:

1) An exception in a finalizer is logged (not by our code; it's logged
as a CLR error, effectively)
2) Throwing an exception is logged by our infrastructure code
3) Running out of memory just throws an exception
4) Calling Environment.Exit reproduces the behaviour - but we don't
have that in our code.

The only feasible explanation I currently have is that some buggy
third-party library code has a call to Environment.Exit in it - but
that seems pretty unlikely, and doesn't properly explain the time
delay, unless it's occurring in a finalizer which somehow didn't get
executed for over 4 hours in one case (gen 2 is feasible, I guess).

Anyone seen anything like this? Hints? Oh, by the way I can't reproduce
this on a test system... ;(

 

[Lasse Vågsæther Karlsen]

Thought I'd throw this out and see if anyone's seen anything similar.

I have a Windows service written in C#, which occasionally stops
silently. The process goes away, Windows notices and shows it as
unstarted in the service control manager.

This always happens (as far as we've seen) after another specific kind
of error (an external resource being unavailable), but the time between
the previous error and the service failing can be hours. The previous
error is logged nicely, but there's nothing in the log just before the
service dies.

I've tried reproducing this by writing deliberately bad code, and seen
the following:

1) An exception in a finalizer is logged (not by our code; it's logged
as a CLR error, effectively)
2) Throwing an exception is logged by our infrastructure code
3) Running out of memory just throws an exception
4) Calling Environment.Exit reproduces the behaviour - but we don't
have that in our code.

The only feasible explanation I currently have is that some buggy
third-party library code has a call to Environment.Exit in it - but
that seems pretty unlikely, and doesn't properly explain the time
delay, unless it's occurring in a finalizer which somehow didn't get
executed for over 4 hours in one case (gen 2 is feasible, I guess).

Anyone seen anything like this? Hints? Oh, by the way I can't reproduce
this on a test system... ;(

Stack overflow? I think one of the ways stack overflows manifests
themselves is that there is no more room to even pop up a message box,
nor would it perhaps be room to call a logging method at that point.
We've certainly seen our share of disappearing apps due to stack
overflows so that was my first reaction, where no hint was left anywhere
of the fact that the application was just moments ago running fine, and
now apparently has crashed.

Could be a red herring though...

 

[Peter Ritchie [C# MVP]]

Are any of your 3rd party libraries native or doing anything natively?

There's rare situations where a stack overflow cannot be handled and Windows
will simply silently terminate the application. I don't think that can
happen with the managed stack...

 

[Jon Skeet [C# MVP]]

Are any of your 3rd party libraries native or doing anything natively?

Wouldn't like to say for sure, to be honest - but quite possibly.

There's rare situations where a stack overflow cannot be handled and Windows
will simply silently terminate the application. I don't think that can
happen with the managed stack...

Right. Both you and Lasse have pointed at the stack, which is a good
point. I'll see what I can provoke along those lines (although probably
sticking within managed code for the moment).

Another thing I'm going to try is deadlocking the finalizer thread.
Given the timing, I reckon there's a finalizer involved *somewhere*...

 

[Darren]

If there is 3rd party unmanaged code it may be calling Win32 SetErrorMode

 

[Willy Denoyette [MVP]]

Thought I'd throw this out and see if anyone's seen anything similar.

I have a Windows service written in C#, which occasionally stops
silently. The process goes away, Windows notices and shows it as
unstarted in the service control manager.

This always happens (as far as we've seen) after another specific kind
of error (an external resource being unavailable), but the time between
the previous error and the service failing can be hours. The previous
error is logged nicely, but there's nothing in the log just before the
service dies.

I've tried reproducing this by writing deliberately bad code, and seen
the following:

1) An exception in a finalizer is logged (not by our code; it's logged
as a CLR error, effectively)
2) Throwing an exception is logged by our infrastructure code
3) Running out of memory just throws an exception
4) Calling Environment.Exit reproduces the behaviour - but we don't
have that in our code.

The only feasible explanation I currently have is that some buggy
third-party library code has a call to Environment.Exit in it - but
that seems pretty unlikely, and doesn't properly explain the time
delay, unless it's occurring in a finalizer which somehow didn't get
executed for over 4 hours in one case (gen 2 is feasible, I guess).

Anyone seen anything like this? Hints? Oh, by the way I can't reproduce
this on a test system... ;(

What do you mean exactly with "unstarted"?, AFAIK there is no such Service
state.

Calling Environment.Exit(..), from within a service should set the Service
state to stopped with a Win32 exit code = 1067.
Also (on XP and higher) you should see an Error message in the System log
with Event ID=7034.

Willy.

 

[Jon Skeet [C# MVP]]

What do you mean exactly with "unstarted"?, AFAIK there is no such Service
state.

Sorry, I was being imprecise. No status is shown, and it can be
started. No doubt this is the Stopped state.

Calling Environment.Exit(..), from within a service should set the Service
state to stopped with a Win32 exit code = 1067.
Also (on XP and higher) you should see an Error message in the System log
with Event ID=7034.

Not in this case - and not in my test service either.

 

[Willy Denoyette [MVP]]

Sorry, I was being imprecise. No status is shown, and it can be
started. No doubt this is the Stopped state.


Not in this case - and not in my test service either.

Weird, what OS are you running?
The normal behavior is as such that when a Service Process dies, that the
SCM logs this as an Error event in the System Eventlog.
When this happens, the SCM also sets the process exit code to 1067
(ERROR_PROCESS_ABORTED) which means: The process terminated unexpectedly.
Every Windows Service process is connected with the SCM via a named pipe IPC
channel, the SCM considers the Service as being dead, whenever the NP
end-point gets closed at the service side, this without being ordered by the
SCM.

Willy.

 

[Jon Skeet [C# MVP]]

Weird, what OS are you running?

XP SP2 on my dev box. Not sure about the servers.

The normal behavior is as such that when a Service Process dies, that the
SCM logs this as an Error event in the System Eventlog.

Hmm. I'll double check tomorrow, but I'm pretty sure that I wasn't
seeing anything. Of course, it's perfectly possible I was looking in
the wrong event log, in which case I apologise for wasting everyone's
time! (Whichever log I was looking in did get various bits of
information, but not this one.)

 

[Jon Skeet [C# MVP]]

Weird, what OS are you running?
The normal behavior is as such that when a Service Process dies, that the
SCM logs this as an Error event in the System Eventlog.
When this happens, the SCM also sets the process exit code to 1067
(ERROR_PROCESS_ABORTED) which means: The process terminated unexpectedly.
Every Windows Service process is connected with the SCM via a named pipe IPC
channel, the SCM considers the Service as being dead, whenever the NP
end-point gets closed at the service side, this without being ordered by the
SCM.

Just checked, and on my dev box if the service uses Environment.Exit we
do indeed get an event in the System event log. Unfortunately on the
server where we're seeing the odd behaviour, there's nothing in the
event log at all around the time of death

 

[Willy Denoyette [MVP]]

Just checked, and on my dev box if the service uses Environment.Exit we
do indeed get an event in the System event log. Unfortunately on the
server where we're seeing the odd behaviour, there's nothing in the
event log at all around the time of death

That means that Environment.Exit was most certainly not called. I just
checked this on XP SP2, W2K3 SP2, Vista and WS2008, and I can say that the
behavior of E.E is consistent across these OS versions.
It's pretty weird in your case (on the server), normally the SCM logs this
message whenever he finds a Service process going away without being
*notified*, for whatever reason, even killing the service process cannot
prevent this. What puzzles me, is that, every Stopped<-> Running transition
is also automatically logged as 'Informational' in the Application event,
however, you don't see such messages, what makes me believe that the service
did not stop itself .....

Willy.

 

[Jon Skeet [C# MVP]]

That means that Environment.Exit was most certainly not called. I just
checked this on XP SP2, W2K3 SP2, Vista and WS2008, and I can say that the
behavior of E.E is consistent across these OS versions.

Fair enough - it seemed unlikely that a library would include a call
to Environment.Exit anyway.

It's pretty weird in your case (on the server), normally the SCM logs this
message whenever he finds a Service process going away without being
*notified*, for whatever reason, even killing the service process cannot
prevent this. What puzzles me, is that, every Stopped<-> Running transition
is also automatically logged as 'Informational' in the Application event,
however, you don't see such messages, what makes me believe that the service
did not stop itself .....

Exactly. It's completely bizarre. I'd expect an orderly shutdown to
log an Application event log message, and a suicide to log in the
System event log. To not get anything is just weird... and very hard
to diagnose, given that we can't reproduce it

Jon

 

[Willy Denoyette [MVP]]

Fair enough - it seemed unlikely that a library would include a call
to Environment.Exit anyway.


Exactly. It's completely bizarre. I'd expect an orderly shutdown to
log an Application event log message, and a suicide to log in the
System event log. To not get anything is just weird... and very hard
to diagnose, given that we can't reproduce it

Hard but not impossible, some techniques are more elegant than the other,
all depends on the version of the OS.
What OS version is there sitting on this server?

Willy.

 

[Jon Skeet [C# MVP]]

Hard but not impossible, some techniques are more elegant than the other,
all depends on the version of the OS.
What OS version is there sitting on this server?

2K3. Oh, and I don't have easy access to it...