How to limit HttpWebRequest connections?

[Basel]

Hi All,

I'm opening a number of threads, and each thread generates http
requests using HttpWebRequest.
How to make each thread open only 1 connection ?
I tried to set a unique ConnectionGroupName for each thread, but after
some time the number of connections exceeds the number of threads (I
see more than thousand connection while only 100 threads are running).

Thank you

 

[Markus Stoeger]

Hi All,

I'm opening a number of threads, and each thread generates http
requests using HttpWebRequest.
How to make each thread open only 1 connection ?
I tried to set a unique ConnectionGroupName for each thread, but after
some time the number of connections exceeds the number of threads (I
see more than thousand connection while only 100 threads are running).

HttpWebResponses are IDisposable. Are you calling Close, Dispose or are
you using "using" on it? That should kill all remaining connections.

hth,
Max

 

[Basel]

I'm closing the stream of the response GetResponseStream()
Anyway, I set :
ServicePointManager.DefaultConnectionLimit=10;

the program stars with 10 connections, but 10 minutes later, it starts
to open and close connections (and I see that there are thousand of
established connections).

is there a way to REALLY limit the code not to exceed a predefined
number of connections???

 

[Nicholas Paldino [.NET/C# MVP]]

Basel,

Setting the connection group is one way to do it, but you aren't
properly disposing of the request. Just because you dispose on the response
stream doesn't mean that you are properly disposing of all the resources
that the request is using.

For any WebRequest derived class (including
HttpWebRequest/HttpWebResponse), you need to dispose of the following:

Stream returned from GetRequestStream on WebRequest
Stream returned from GetResponseStream on WebResponse
WebResponse (it implements IDisposable)

Hope this helps.

 

[Basel]

I closed the streams,

I set this:
ServicePointManager.DefaultConnectionLimit=10;
ServicePointManager.MaxServicePointIdleTime=10000;

when I run the following code in threads, after a couple of minutes I
see in the network status that hunderds of connections are opened, and
some connections are closing.

here is the code:

for(int i=0;i<1000000;i++)
{
HttpWebRequest request =
(HttpWebRequest)WebRequest.Create("http://myserver");

using(WebResponse response = request.GetResponse())
{

StreamReader reader = new
StreamReader(response.GetResponseStream());

string str = reader.ReadLine();
//do some parsing on str
response.GetResponseStream().Close();

}
}

Note: I'm running 2000 request/ second

The code is ok? or I missed something!?

Thank you!

 

[Guest]

Basel,
No this is probably not alright. What you are doing here is opening a large
number of Requests all on the same thread, in a serial manner. This can not
only take a long time, but it is fraught with complications.

You want to structure your application to do this either on the default .NET
threadpool so you are running each on a background thread with an
asynchronous callback method to get the response, get your data and clean up,
or you want to use a custom threadpool. In this manner as each connection is
processed and disposed, that threadpool thread is freed up to handle another
workitem.
Peter

 

[Basel]

My goal is to open only ONE connection for each thread that should stay
active along the way.

BTW, the more ServicePointManager.MaxServicePointIdleTime is smaller,
connections grows more rapidly

It seems all the threads uses only one connection from the established
connections. when the timeout occurs it renews the timed-out
connections.

What is the correct way to force each thread open one *socket* that
will be used only by this thread?

 

[Joerg Jooss]

Thus wrote Basel,

Hi All,

I'm opening a number of threads, and each thread generates http
requests using HttpWebRequest.
How to make each thread open only 1 connection ?
I tried to set a unique ConnectionGroupName for each thread, but after
some time the number of connections exceeds the number of threads (I
see more than thousand connection while only 100 threads are running).

Are you sure closing the connection isn't initiated by the server?

Cheers,

 

[Basel]

I dont know if closing the connection is initiated by the serve.

Let let me ask a question,
I want to create 10 threads so that each thread runs a loop that sends
a request to server and gets the response.

and I want each thread to open a single connection.

Can you please tell me how to do that?

 

[Joerg Jooss]

Thus wrote Basel,

I dont know if closing the connection is initiated by the serve.

Well, find out ;-)

Let let me ask a question,
I want to create 10 threads so that each thread runs a loop that sends
a request to server and gets the response.
and I want each thread to open a single connection.

That's not possible, unless both client and server agree to use a persistent
connection. While this is the default behaviour in HTTP 1.1 and thus HttpWebRequest,
it doesn't mean that your server is willing to keep the connection open.
There's no point in reusing a TCP connection if your server sends a "Connection:
close" or switches back to HTTP 1.0's default behaviour -- the server will
close the connection and not accept any further requests on it.

Can you please tell me how to do that?

Monitor the HTTP traffic between your client and your server with a network
sniffer or a web proxy to see whether your server closes connections.

Cheers,

 

[Greg Young]

Can't see the rest of this thread but you may also want to take a look at
the ServicePoint class
http://msdn2.microsoft.com/en-US/library/system.net.servicepoint.aspx

Cheers,

Greg Young