Richard said:
FYI - I have a very similiar scenario to yours; I'm running a C# service on
a 2 HT Zeon processor box. I have a bunch of custom perfmon counters in code
and I know that ThreadPool worker thread counts stay right around 100 for me;
I dunno what IO completion thread counts are as I haven't had any issues with
them. Be aware that the system often needs a 'shadow thread' for every
thread you spin off so high thread counts in and of themselves are not always
an issue...
How exactly are you using asynch? Fire and forget or BeginReceive and
sleep? In my case my service receives 3 seperate pieces of data serially
before acting on the data, so I do 3 BeginReceive/sleep's in a row with my IO
completion handler setting the IO completion event each time. In each case I
sleep on IO completion and/or a service command event. This is working very
well for me.
Well, I guess it's more or less 'fire and forget' in that I initiate a
BeginReceive/BeginSend and then have functions to handle the
completion of the async call (i.e. via EndReceive/EndSend).
Also be aware that your IO completion handler should catch BOTH
SocketException and generic Exception --> if the socket is closed by a worker
thread within your code {aka if you shutdown your program or service and some
other worker thread closes the socket} the IO completion callback will fire
with a generic Exception and a message that reads "socket disposed" aka:
// Called by ThreadPool IO completion thread
void ReceiveIOCompletionCallback(IAsyncResult ar)
{
try
{
// wait for IO to complete then signal BeginReceive caller thread
((Socket)ar.AsyncState).EndReceive(ar);
((ManualResetEvent)ar.AsyncWaitHandle).Set();
}
catch (SocketException ex)
{
// get here if remote client closes socket or other expected error...
}
catch (Exception ex)
{
// get here if another of your threads disposes socket
}
}
Check. I've got those catches in my code.
If you are seeing runaway thread counts it could be several things. Are
they IO completion threads or are they worker threads?
I'm assuming IO completion threads. I'm not using the ThreadPool
explicitly. I spawn a very well-defined number of Thread objects
explicitly for very specific work that goes on in the background
throughout the lifetime of the process. For all async socket comm, I
let the CLR do its magic...
If they are IO completion threads then you might be ok. In the IO
completion callback code above an IO completion thread is active and it
counts against the process as an active thread - the thread sleeps until the
IO completes but it will count as an active thread. Your high thread counts
might just mean that you have many IO requests pending and that there are
many IO completion threads sleeping while IO is in progress.
-OR- You could be getting a socket error or partial IO completion and many
IO completion threads are active to service the completed operations. If you
are doing partial read/write operations then you could see high thread counts
if operations are partially completing. In my stuff I don't allow partial IO
completion - I begin a write/read operation and sleep until the IO fully
completes.
On the other hand if your high thread counts are due to your worker threads
then invariably you have an issue with the logic related to spinning off a
thread - some condition is triggering your logic to spin off many threads -
you'll probably want to fix that...
Yeah, no worker threads (i.e. via ThreadPool) in my code, so I'm
pretty sure it's something in the IOCP arena...
How long until the automagic brings thread counts back down? A few seconds
or something along the lines of a socket timeout?
When it's happened, the duration has been inconsistent, but certainly
significantly longer than a socket timeout (i.e. > 5-10 min sometimes)
Why only one at a time? One great advantage of asynch IO is having many
things happening at the same time. You can queue up several reads and
writes, they are guaranteed to be serviced in the order in which they are
queued...
In my stuff I have coded a 'waitable semaphore' as a throttling mechanism.
All socket activity is throttled by the configurable maximum number of
threads the semaphore will permit. At different points in code threads wait
for IO completion, a semaphore reference, and/or a service command or any
arbitrary combination of those events...
--Richard
Well, I think this implementation is sort of analogous to the code
snippet you've provided above. Basically, I have a thread that goes
through a queue and pulls out data that needs to be forwarded, via a
socket connection, to one of many connected clients. Thread takes the
data and initiates an async send and then goes back to the top of its
loop without waiting for the send to complete (thus, async). If the
thread pulls data off the queue that needs to be sent to a client that
is currently in the process of receiving data (asynchronously), it
defers that send. This is done to try and avoid the 'pile-up'
situation with a huge number of sends pending... (and the ensuing high
thread count associated with it, context switching costs, etc..)
To note, the 'processing' threads that I explicitly spawn are assigned
a Highest priority (this is due to the real-time nature of the
server.. lags/delays are viewed by users very poorly). So with that
being the case, I've added Thread.Sleep(0) directives at the end of
each of my explicitly-spawned Thread's loops and am testing the impact
of that today in the beta environment. The reason I added this is
because I've theorized that my high-priority threads are starving the
IOCP threads (which, I'm assuming, are of a 'normal' priority) and
barring them from invoking their callbacks and, in high traffic
periods, this results in the thread count build up I've observed...
I went out on the newsgroups and there seemed to be some confusion as
to whether Thread.Sleep(0) would have any impact if the statement were
executed by a thread with a high priority vis-a-vis other
lower-priority threads- in that because other threads had a lower
priority, the Sleep(0) invocation would do nothing at all and the
high-priority thread would keep on running. I personally don't know
the definitive answer to this, yet.. continuing to investigate. I also
changed the base priority on the process itself (to High). Not sure of
the _exact_ implications of this on thread scheduling, etc...
Anyway, thanks for the comments and suggestions. I'll see how today's
server run goes and post a follow-up. If you have any clarification on
the whole thread prioritization/scheduling issue, that would be much
appreciated as well!
