Configuring More Than Two HttpWebRequests To The Same Host?

[Alexander J. Oss]

I'm writing a C# application for .NET 2.0 in which I intend a number of
background worker threads to be conveying information from our SQL Server
database to a legacy database via HTTP GET requests. Each worker thread
runs the following code in a loop:

-----
HttpWebRequest request = (HttpWebRequest)WebRequest.Create([snip]);

HttpWebResponse response = (HttpWebResponse)request.GetResponse();

Stream responseStream = response.GetResponseStream();
StreamReader responseReader = new StreamReader(responseStream);
string responseText = String.Empty;
string line;
while ((line = responseReader.ReadLine()) != null)
{
if (line != String.Empty)
responseText += line;
}

responseReader.Close();
response.Close();
-----

The total number of requests will end up being approximately 30,000 for each
run of the transfer, so I'm trying to run as many threads as I can so the
total run time will be reduced. But it appears that .NET by default
restricts the number of simultaneous network connections to a single host to
2 (why?!). I did find that if I create an application configuration file
that looks like this:

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
<system.net>
<connectionManagement>
<add address = "10.134.0.2" maxconnection = "24" />
<add address = "*" maxconnection = "2" />
</connectionManagement>
</system.net>
</configuration>

I should be able to have up to 24 connections to that 10.134.0.2 host (see
http://support.microsoft.com/kb/821268/en-us).

However, this isn't working--I still seem to be constrained to two
connections. Any suggestions as to what I might be doing wrong? The
application name is RightsTransferToADAMS.exe and I have a
RightsTransferToADAMS.exe.config file in the same directory with the content
above. Is there a way to programmatically check that the config file has
been loaded correctly and that I should be able to make 24 connections? Do
I need to do something else?

Thanks very much for any help you might be able to provide.

 

[Greg Young]

Take a look at the ServicePoint class
http://msdn2.microsoft.com/en-us/system.net.servicepoint.aspx

http://msdn2.microsoft.com/en-us/library/7af54za5.aspx includes a bit more
on it.

Cheers,

Greg Young
MVP - C#
http://codebetter.com/blogs/gregyoung

 

[Guest]

Alexander,
If you use a ThreadPool and the asynchronous pattern with async callback
methods, you should be able to make as many connections as you need (within
reason).
Peter

 

[Alexander J. Oss]

Okay, here's what I ended up doing. I'm sure there's redundancy in there,
but I'm not taking the time to experiment with which things are necessary or
not at the moment. Besides... it turns out that with only a few threads
(5-10) I can max out our old HP 9000 D270's two CPUs (it's a CGI app that
kicks off a Progress database client process with each call... horribly
inefficient).

1. Created the application configuration file I had in the earlier post.

2. Added this line of code before creating any HttpWebRequest objects:

ServicePointManager.DefaultConnectionLimit = 24;

3. Added these lines of code for each HttpWebRequest object created:

request.KeepAlive = false;
request.ServicePoint.ConnectionLimit = 24;
request.ProtocolVersion = HttpVersion.Version10;

Thanks again for everyone's help!