Am I using ThreadPool the right way?

[Ricardo Vazquez]

Hello everybody,

I'm programming a TCP server.
During the stress tests I noticed that maybe my socket-receiving thread
became deaf after an hour of heavy stress.
I think that the reason could be this: The processing of the received
messages. Something goes wrong in this processing and the code gets stuck
here. As the processing is within the receiving "while" loop, the loop gets
also stuck, so that we will never reach again the Socket.Receive invoke;
thereafter we are deaf.

Besides I disliked the idea of processing the message within the receiving
loop. Somehow I understand that receving should be more "disconnected" from
the rest of the code.

So I thought of a different thread, asynchronously processing the received
message.
And, as after heavy stress this thread could get stuck as I said, I thought
of using the ThreadPool (even though one processing-thread gets stuck, the
following messages will be processed anyway, because ThreadPool will give us
a new thread to do this).

So, code would be something like this:
------------------------
int nRes = 0;
Byte[] buffer = new Byte[MAX_RECV_SIZE];
try
{
nRes = m_pSocket.Receive(buffer, MAX_RECV_SIZE, SocketFlags.None);
}
catch ...
....
string msg = Encoding.ASCII.GetString(buffer, 0, nRes);
sLog = String.Format("[pbx] CTIServer {0} < recv: '{1}'",
getPbxStringId(), msg);
m_maq.log.logNormal(3, sLog);
try
{
ThreadPool.QueueUserWorkItem(new WaitCallback(processingThread), msg);
}
catch (ApplicationException appEx)
{
sLog = String.Format("[pbx] Exception thrown while creating received
buffer processing thread: " + appEx.Message);
m_maq.log.logWarning(1, sLog);
}
catch (Exception)
{
sLog = String.Format("[pbx] Generic exception thrown while creating
received buffer processing thread");
m_maq.log.logWarning(1, sLog);
}
------------------------

It works just fine, and I think that this way I achieve my goal: receiving
and processing are not coupled.

BUT it is the first time I use the ThreadPool, and I'm quite new to C# (I'm
arriving from C++), so I would like to ask you if you think I'm using
ThreadPool the right way, if you think that this usage could end up with
some other problem or serious inconvenience, and to ask you for hints and
advices.

Thank you!

Ricardo Vázquez.
Madrid, Spain.

 

[Ignacio Machin \( .NET/ C# MVP \)]

Hi,


--
Ignacio Machin
http://www.laceupsolutions.com
Mobile & warehouse Solutions.

Hello everybody,

I'm programming a TCP server.
During the stress tests I noticed that maybe my socket-receiving thread
became deaf after an hour of heavy stress.

You should have a thread just to receive new connections, once a connection
is received a new thread is created and that thread is the one that handle
that connection. This allows you to receive new incomming calls.

The "worker thread" can either use async or sync request. I think it depends
of what are you transfering.

I think that the reason could be this: The processing of the received
messages. Something goes wrong in this processing and the code gets stuck
here. As the processing is within the receiving "while" loop, the loop
gets
also stuck, so that we will never reach again the Socket.Receive invoke;
thereafter we are deaf.

Using the above you will never get deaf. The server will always answer.

Besides I disliked the idea of processing the message within the receiving
loop. Somehow I understand that receving should be more "disconnected"
from
the rest of the code.

It depends, in my code the receiving is sync and therefore is inside the
loop. You can decoople it if your scenario allows it.
Just take into account that it will make the code more difficult to
maintain/read.

 

[Ricardo Vazquez]

Hi Ignacio!

Thank you very much for your advice and interest!

Kind regards,

Ricardo.

You should have a thread just to receive new connections, once a
connection is received a new thread is created and that thread is the one
that handle that connection. This allows you to receive new incomming
calls.

The "worker thread" can either use async or sync request. I think it
depends of what are you transfering.

You are right, of course. But I already have this:
A thread listening for connections.
When a new client connects I get a new socket, and I create a new client
object with it; somthing like this:
------------------------------------
while(!m_stopServer)
{
try
{
clientSocket = m_server.AcceptSocket();
socketListener = new ClientThread(m_maq, clientSocket);
lock(clients.SyncRoot)
{
clients[socketListener.ClientSocket] = socketListener;
}
socketListener.StartSocketListener();
}
catch ... etc.
...
}
------------------------------------
But, as you see, the new client starts a receiving thread:
socketListener.StartSocketListener().
This is the thread I was talking about, the one that got stuck after one
hour of stress testing, the one I was trying to de-couple: the
client-instance receiving thread.

 

[Peter Duniho]

Hello everybody,

I'm programming a TCP server.
During the stress tests I noticed that maybe my socket-receiving thread
became deaf after an hour of heavy stress.
I think that the reason could be this: The processing of the received
messages. Something goes wrong in this processing and the code gets stuck
here. As the processing is within the receiving "while" loop, the loop gets
also stuck, so that we will never reach again the Socket.Receive invoke;
thereafter we are deaf.

Besides I disliked the idea of processing the message within the receiving
loop. Somehow I understand that receving should be more "disconnected" from
the rest of the code.

I think generally that is a good goal, yes.

You really need to post a concise-but-complete sample of code that
reliably reproduces your problem. Showing excerpts is not very useful,
as there's no way to know whether the excerpt includes the problematic
part or not.

On the other hand, posting your complete code is also not very useful,
as that usually results in there being way too much code for someone not
being paid to debug your code to inspect. Thus "concise". Post only
the bare minimum required to reproduce the problem.

Noting, of course, that with networking code there are two ends to every
communication, and you must post both ends for the sample to be truly
usable. A complete sample for one end is better than nothing, but it
won't allow anyone to actually compile and run your code to see what's
going wrong.

Now, all that said: I don't see how using the ThreadPool will help this.
There's a maximum number of threads that the ThreadPool will have, and
so if you've got a bug in which your code is somehow getting stuck, you
will eventually consume all of the ThreadPool threads, as those get
stuck one by one instead of your receiving thread. All you do by using
the ThreadPool is move the problem somewhere else.

It's not possible to say from your post why your code gets stuck.
However, if you are using multiple threads it's likely that you've got a
deadlock bug somewhere. You should use the debugger when your program
gets stuck and see where in each thread it is waiting. If it's a
deadlock problem, you'll have at least two threads waiting on something
that is either directly or indirectly held by the other thread.

You should also consider using the async methods for the Socket class.
If you are looking to take advantage of threading to get the i/o out
into its own code path, the async methods are a great way to accomplish
that. They aren't necessarily going to solve your deadlock problem, but
they will clean up the overall design, and possibly make it easier to
figure out what is deadlocking.

Pete

 

[Chris Mullins [MVP - C#]]

The reality is that you're not really using the ThreadPool the right way.
For a client app, you would be fine. But for a high-load server, it's not
going to work quite right.

Under stress, you're going to get Thread Pool starvation, and your
application will deadlock.

For a set of very detailed explinations and some suggestions, see:
http://www.coversant.com/Coversant/Blogs/tabid/88/EntryID/8/Default.aspx
http://www.coversant.com/Coversant/Blogs/tabid/88/EntryID/10/Default.aspx

--
Chris Mullins

Hello everybody,

I'm programming a TCP server.
During the stress tests I noticed that maybe my socket-receiving thread
became deaf after an hour of heavy stress.
I think that the reason could be this: The processing of the received
messages. Something goes wrong in this processing and the code gets stuck
here. As the processing is within the receiving "while" loop, the loop
gets
also stuck, so that we will never reach again the Socket.Receive invoke;
thereafter we are deaf.

Besides I disliked the idea of processing the message within the receiving
loop. Somehow I understand that receving should be more "disconnected"
from
the rest of the code.

So I thought of a different thread, asynchronously processing the received
message.
And, as after heavy stress this thread could get stuck as I said, I
thought
of using the ThreadPool (even though one processing-thread gets stuck, the
following messages will be processed anyway, because ThreadPool will give
us
a new thread to do this).

So, code would be something like this:
------------------------
int nRes = 0;
Byte[] buffer = new Byte[MAX_RECV_SIZE];
try
{
nRes = m_pSocket.Receive(buffer, MAX_RECV_SIZE, SocketFlags.None);
}
catch ...
...
string msg = Encoding.ASCII.GetString(buffer, 0, nRes);
sLog = String.Format("[pbx] CTIServer {0} < recv: '{1}'",
getPbxStringId(), msg);
m_maq.log.logNormal(3, sLog);
try
{
ThreadPool.QueueUserWorkItem(new WaitCallback(processingThread), msg);
}
catch (ApplicationException appEx)
{
sLog = String.Format("[pbx] Exception thrown while creating received
buffer processing thread: " + appEx.Message);
m_maq.log.logWarning(1, sLog);
}
catch (Exception)
{
sLog = String.Format("[pbx] Generic exception thrown while creating
received buffer processing thread");
m_maq.log.logWarning(1, sLog);
}
------------------------

It works just fine, and I think that this way I achieve my goal: receiving
and processing are not coupled.

BUT it is the first time I use the ThreadPool, and I'm quite new to C#
(I'm
arriving from C++), so I would like to ask you if you think I'm using
ThreadPool the right way, if you think that this usage could end up with
some other problem or serious inconvenience, and to ask you for hints and
advices.

Thank you!

Ricardo Vázquez.
Madrid, Spain.

 

[Ricardo Vazquez]

Hi Pete,

Thank you very much for your answer.
You are right: If there is really a deadlock, the usage of 1 thread or 25
threads is not a solution: they will all finally get stuck and nothing was
solved.
I need to debug that possible deadlock, as you said: "You should use the
debugger when your program gets stuck and see where in each thread it is
waiting. If it's a deadlock problem, you'll have at least two threads
waiting on something that is either directly or indirectly held by the other
thread."

But, as I said, I'm new to .net and Visual Studio 2005...

What is it the way to debug a deadlock? Any guidelines, web links,
orientation, detailed explanation...?

Once again, thank you very much!

Ricardo.



Advanced Telephony
Applications --------------------------------------------------------------------------------
[Texto] Ricardo Vázquez Almagro Jusan, s.a. Research & Development Tel: +34
91 456 0110 (ext. 52) Fax: +34 91 553 1411 mailto: (e-mail address removed) web:
www.jusan.es

 

[Peter Duniho]

[...]
But, as I said, I'm new to .net and Visual Studio 2005...

What is it the way to debug a deadlock? Any guidelines, web links,
orientation, detailed explanation...?

First, you'll need to be using a retail version of Visual Studio. It is
possible to debug deadlocks and other threading issues in the Express
version, but it's not something I'd recommend for someone unfamiliar
with the general techniques of dealing with thread issues in the first
place, since Express doesn't provide any direct way to get at the
individual threads in the debugger.

So, assuming you have a retail version of VS, the next step is to run
your application under the debugger until it gets stuck (or attach the
VS debugger to a process running your application after it gets stuck).

At that point, use the Break command to stop execution of your program.
Once the process has been interrupted, you can then use the Threads
drop-down in the Debug toolbar to examine the state of each thread. It
will allow you to switch from one thread to another, with the source
view, stack trace, local variables, etc. being updated according to the
current thread.

Some of the thread will be irrelevant, and may not even include any of
your own code (all of the code in the stack trace will be "external").
You'll be looking for threads that are stopped on a statement that waits
for some resource, to identify which threads are involved in the
deadlock and why they are waiting.

At some point, you should come across at least two threads that are
waiting, each of which are waiting on a different resource, but in each
case a resource being held by some other thread that is also waiting.
That's your deadlock.

Once you've identified that, then you need to figure out what in your
design led to the bug where you have that circular resource acquisition
going on and fix that. Depending on your experience level that in and
of itself could be a challenge, but regardless you need to first
identify the problem for that to even be an option.

Pete