Threading Question

[Gary Short]

Hello group,

I'm writing a spidering app and I'd appreciate some advice on
threading. My plan of attack is to have a main thread fetching the
html from a list of urls in a xml file, then spawning a new thread
using ThreadPool.QueueUserWorkItem(new WaitCallback(threadFunction));
to do the actual indexing in an asynchronous manner.

My understanding is that this method will cause the function to be
queued for execution by the next available ThreadPool thread. Now my
question is, can I just keep calling this method ad infinitum without
having any adverse effect on resources, this all being handled by the
ThreadPool, or is there a maximum number of threads I should be
spawning?

Thanks in advance,
Gary

 

[Zürcher See]

I'm not sure, but i think by default there are 25 threads in the threadpool,
you will just queue the function ,wenn a thread is free will get the next
job, but I think you can change the maximal number of thread in the thread
pool

 

[David Browne]

Hello group,

I'm writing a spidering app and I'd appreciate some advice on
threading. My plan of attack is to have a main thread fetching the
html from a list of urls in a xml file, then spawning a new thread
using ThreadPool.QueueUserWorkItem(new WaitCallback(threadFunction));
to do the actual indexing in an asynchronous manner.

My understanding is that this method will cause the function to be
queued for execution by the next available ThreadPool thread. Now my
question is, can I just keep calling this method ad infinitum without
having any adverse effect on resources, this all being handled by the
ThreadPool, or is there a maximum number of threads I should be
spawning?

Queuing a workitem does not spawn a thread. So you don't really have to
worry there. But you might not want to queue all the work at once. The
ThreadPool offers GetAvailableThreads, GetMaxThreads, GetMinThreads for you
to monitor activity in the ThreadPool. Plus you can use callbacks and
global counters from you threadFunction to track the progress of the work.

You might not want to queue additional workitems while there are more than a
certian number of executing threads. In general with a threadpool, your
overall throughput goes up as you add worker threads for a while, then
levels off, and then goes down. Depending on your machine and the kind of
work, you may get the best throughput with 2,4, or 8 concurrent threads.
The point it, 25 may be too many.

David

David

 

[Alvin Bruney]

25 threads per processor actually

 

[Gary Short]

Thanks for the advice David, I think I'll make the number of threads
user configurable, but limit it to say 10 or 15. I believe the 25
thread per processor limit has been raised in the 1.1 version to
something like 1,000.

Cheers,
Gary