2003 / XP multi-threaded problems

[Rob Durant]

Hi,

I have a multi-threaded application (have also tried as service - same
behaviour) that runs fine on XP, but not on 2003. Symptoms are:

Threads are started normally, locks acquired and released as needed,
runs fine for some time, then stops - Visual Studio detects no
exceptions, and I have placed break points on all the handling code to
catch them. No errors / failures are logged, but child threads have all
exited, and I am left with a single thread of the application or service
instance. There are no event log entries either. The error happens
regardless of whether the debugger is attached.

I have spent a lot of time debugging this and I'm running out of
thoughts. The test server (2003) is fully patched to the same point as
my XP development machine.

The application / service is completely in managed vb.net code.

Any help would be greatly appreciated

Rob

 

[Rob Durant]

Found out some more information...

The test machine is slower (considerably) than the development machine
and it not multi-threaded. When I try and start too many threads, the
application gets to a point where it just ditches the threads with out
seeming to raise an error ...

e.g.

thread 1 starts instances of thread 2 each time it needs to write to a
database. thread 2 for some reason is delayed in it's completion (for
example if connection to db is broken) - this causes threads to stack
up. When thread 1 has reached a critical mass, it just dumps all
instances of thread 2 and sits there pretending nothing happened. The
work done on thread2 is critical to the application, so I don't want to
just timeout on the thread.

Can anyone tell me if the threads raise an event of some kind and if so
how should I catch it (no exceptions are thrown)

Thanks

Rob

 

[Guest]

Rob,

Found out some more information...

The test machine is slower (considerably) than the development machine
and it not multi-threaded. When I try and start too many threads, the
application gets to a point where it just ditches the threads with out
seeming to raise an error ...

e.g.

thread 1 starts instances of thread 2 each time it needs to write to a
database. thread 2 for some reason is delayed in it's completion (for
example if connection to db is broken) - this causes threads to stack
up. When thread 1 has reached a critical mass, it just dumps all
instances of thread 2 and sits there pretending nothing happened. The
work done on thread2 is critical to the application, so I don't want to
just timeout on the thread.

Can anyone tell me if the threads raise an event of some kind and if so
how should I catch it (no exceptions are thrown)

I think you either have a deadlock issue, or you spawn too much threads. It
is typical that when software is put on a faster/slower machine, or a machine
with fewer/more CPU's, it exposes multi-threading problems that were already
there but didn't trigger. Another thing I'd advise against, is having threads
with different priorities; this could induce dirty deadlocks (ie hard to
find).

I don't understand you when you say 'threads stack up'. What do you mean
here, that they just wait for db actions to complete? That is a regular
waiting operation and I don't see how that could lock it up completely (a
Thread.Interrupt should be able to unblock it).

What do you mean when you say 'thread 1 reaches critical mass'??? What do
you mean by 'all instances of thread 2'. You mean you have a specific
function executed in several threads?

Kind regards,

 

[Guest]

Can anyone tell me if the threads raise an event of some kind and if so

how should I catch it (no exceptions are thrown)

There has been a lot of discussion about threading and exceptions in these
newsgroups. You might try searching for some interesting reading, although I
guess you have already tried that.

Contrary to what I have read, in a multi-threaded windows form app, I think
you need to enable exception handlers for both UnhandledException and
ThreadException. For testing my app, I arrange for an exception to be thrown
in the gui thread or in a new worker thread. When I run in the .net ide with
out of the box exception settings, the gui thread exception is trapped by my
handler, but the worker thread exception is not, but it is trapped in the
ide. When I run outside the ide, both are trapped. The gui thread throws
ThreadException, and the worker thread throws UnhandledException.

Not so intuitive, in my opinion - the choice of names is peculiar, and why
does the ide step in front if my handler? So, what to do? Enable both in
your sub main, and arrange for both to report out info if they are called,
and be suspicious of different behavior in the ide vs standalone.

 

[Rob Durant]

There has been a lot of discussion about threading and exceptions in these
newsgroups. You might try searching for some interesting reading, although I
guess you have already tried that.

Contrary to what I have read, in a multi-threaded windows form app, I think
you need to enable exception handlers for both UnhandledException and
ThreadException. For testing my app, I arrange for an exception to be thrown
in the gui thread or in a new worker thread. When I run in the .net ide with
out of the box exception settings, the gui thread exception is trapped by my
handler, but the worker thread exception is not, but it is trapped in the
ide. When I run outside the ide, both are trapped. The gui thread throws
ThreadException, and the worker thread throws UnhandledException.

Not so intuitive, in my opinion - the choice of names is peculiar, and why
does the ide step in front if my handler? So, what to do? Enable both in
your sub main, and arrange for both to report out info if they are called,
and be suspicious of different behavior in the ide vs standalone.

Thanks for this advice. I have spent another day checking the code and
playing around with it.

By critical mass I meant a defined limit that seemed to be more than the
computer could cope with (although CPU usage was around 4-5% at the time!?!)

I have a manager that spawns 'main jobs', (perhaps 1 to 3 at a time)
which spawn 'sub-jobs' themselves... only 10 instances of the 'sub-jobs'
are allowed at any one time, and I am testing with only one 'main job'
running.

I think I may be closer to the answer...

The main worker thread starts by initialising and starting a timer. It
then uses each elapses event to do some work and set the 'sub job'
going, but by the time an elapsed event occurs on the timer. Once the
work is done, it will use the elapsed event to stop the timer and exit.
But the thread seems to be exiting as soon as the initialisation
function has completed.

So another question...

Is it right to expect that because a timer is running, the thread should
stay alive? or should a thread exit when the function exits despite the
thread still performing work? I would have expected the thread to stay
alive for as long as the timer is running. If the first is true, how
should I keep the function alive at the same time as handling the timer
ticks?

Rob

 

[Guest]

Is it right to expect that because a timer is running, the thread should

stay alive? or should a thread exit when the function exits despite the
thread still performing work?

Ultimately, a thread starts with a call to a sub/function. When that
sub/function exits/returns, the thread is over. Period. It sounds to me
like this is the source of your problem.

What you need to do in general terms is as follows. According to some rules
of your choosing, you launch worker threads. The rules can include whatever
you want. Maybe you want only a couple running at a time. Maybe some depend
on outputs of others. The specifics of that are up to you. But the app's
main thread should be responsible for launching the workers. A timer for the
main thread is likely to be useful to you.

The worker threads should do their work to completion and then quit. As
they produce output, they should make the data available to the main thread.
Here, thread contention issues need to be solved. What I use is a queue -
threads thread-safely append outputs to a queue. The main thread can
periodically sample the queue (alternatives to polling are also possible).

One thing to worry about is launching too many or not enough threads. Say
you want to retrieve 100 urls from the web. You could launch 100 threads,
each responsible for one url. You could launch one thread responsible for
all 100. Or 10 threads each responsible for 10 urls. 100 threads is bad
because it is resource unfriendly. 1 thread is bad because one slow web
server stops the entire download process, ie it is really single threaded.
10 threads is good because it is not too resource unfriendly and you get all
the benefits of multi-threading.

This is fairly easy to arrange in a standard windows form app which has a
message pump and all that entails. Your main thread is the gui thread, and
it is awakened by events, and you can arrange for your worker threads to do
things that cause the gui thread to respond to an event. So, as time passes
and conditions change, your main thread can launch workers, and as the
workers finish their jobs, the main thread sees and responds to their outputs.

I have no in depth experience with services, console apps, and other
settings, so I don't want to make the same 'fairly easy' assertion. But my
guess is that this same general approach should work in these settings.