threads

[Pracownik]

Hello,

I'm writing application in which I'm downloading contents of many web
pages.
I accomplish this in separate threads, each thread to download one
page.
The process of downloading web page is not heavy on processor, it
takes network resources.
What benchmark should i develop in order to determine the maximum
concurrent threads that can work on my machine
for downloading web pages ?

Does anyone has suggestion ?

Thanks
Wojtek

 

[Pracownik]

I recommend not using threads directly.  Instead, use the async API
available for the .NET object(s) you're using for the download.  You'renot
specific about which .NET objects you're using, but all the usual ones --
NetworkStream, Socket, HttpWebRequest, etc. -- do have async versions of
their members.

Using the async API, you allow .NET to handle thread management; on
supported platforms (I don't recall exactly, but pretty much everything
except old legacy Win9x-based OSs), .NET will use IOCP threads to manage
the i/o itself.  This allows a small number of threads to handle a
relatively large number of concurrent i/o operations.

Then you only have to worry about how many different sites you want to
download from at a time.  That in and of itself is a difficult enough
problem, so you should stay focused on it rather than getting distracted by
trying to worry about something that .NET already does well.  

Pete

Thank you for advice, i have implemented the solution and it works
well.