GetRequestStream slows down?

G

Guest

I have a multithreaded application that is posting XML requests to a remote
web site. As more and more calls are made to
HttpWebRequest.GetRequestStream, the length of time increases for each call
from an initial 2-3 seconds to anywhere from 15-30 seconds. What am I doing
that is causing this method call's duration to increase and how do I make it
run faster? Below is some pseudo-code for the method that is called in a
multi-threaded fashion...

- Jim

private void MultithreadedMethod()
{
string url = ConfigurationManager.AppSettings["providerUrl"];

request = (HttpWebRequest)WebRequest.Create(url);
request.Method = "POST";
request.ContentType = "text/xml";

startCount = Environment.TickCount;
using (Stream s = request.GetRequestStream())
{
elapsedCount = Environment.TickCount - startCount;
//the elapsed time increases with each call
EventLog.WriteEntry("request.GetRequestStream() = " + elapsedCount,
EventLogEntryType.Information);

using (StreamWriter writer = new StreamWriter(s))
{
writer.WriteLine(requestXml);
writer.Flush();
writer.Close();
}

s.Close();
}


request.Timeout = requestTimeoutMilliseconds;
response = request.GetResponse();

//handle the response here
}
 
J

John Murray

Are these calls concurrent or consecutive. In 1.1 these requests were
made on the thread pool, so you might be running into some concurrency
issues.

Additionally, depending on the size of the page returned, you might also
be running in to GC issues -- so for example, if the content returned is
over 85k (assuming it being put into one big string variable at some
point) that memory has the potential of being put into the large object
heap which is only swept and not compacted .... so you might be running
into some memory issues.

Also, the s.Close is redundant and repetitive ... it's getting called in
IDispose after you exit from your using block.

Also, make sure that you are closing the response object that you get
back from request.GetResponse(); .... not doing so can lead to some
unmitigated nastiness

John
 
G

Guest

Aside from the other comments, the title of your method "MultithreadedMethod"
is confusing, as it isn't multithreaded at all the way it is written.
If you really have a lot of this going on, you probably want to look at the
asynchronous BeginXXX / EndXXX overloads which use the Threadpool "under the
hood" - or use a custom threadpool.
Peter
 
G

Guest

What I meant by "MultithreadedMethod" is that the method is called many times
on many different threads.

I've seen the asynchronous calls for GetRequestStream... will they get me
around potential contention issues mentioned?

Peter Bromberg said:
Aside from the other comments, the title of your method "MultithreadedMethod"
is confusing, as it isn't multithreaded at all the way it is written.
If you really have a lot of this going on, you probably want to look at the
asynchronous BeginXXX / EndXXX overloads which use the Threadpool "under the
hood" - or use a custom threadpool.
Peter

--
Co-founder, Eggheadcafe.com developer portal:
http://www.eggheadcafe.com
UnBlog:
http://petesbloggerama.blogspot.com




Jim Toth said:
I have a multithreaded application that is posting XML requests to a remote
web site. As more and more calls are made to
HttpWebRequest.GetRequestStream, the length of time increases for each call
from an initial 2-3 seconds to anywhere from 15-30 seconds. What am I doing
that is causing this method call's duration to increase and how do I make it
run faster? Below is some pseudo-code for the method that is called in a
multi-threaded fashion...

- Jim

private void MultithreadedMethod()
{
string url = ConfigurationManager.AppSettings["providerUrl"];

request = (HttpWebRequest)WebRequest.Create(url);
request.Method = "POST";
request.ContentType = "text/xml";

startCount = Environment.TickCount;
using (Stream s = request.GetRequestStream())
{
elapsedCount = Environment.TickCount - startCount;
//the elapsed time increases with each call
EventLog.WriteEntry("request.GetRequestStream() = " + elapsedCount,
EventLogEntryType.Information);

using (StreamWriter writer = new StreamWriter(s))
{
writer.WriteLine(requestXml);
writer.Flush();
writer.Close();
}

s.Close();
}


request.Timeout = requestTimeoutMilliseconds;
response = request.GetResponse();

//handle the response here
}
Click to expand...
Click to expand...
 
G

Guest

Correct, which is a misnomer. If it is called many different times, you
should probably be using the Threadpool and the asynchronous
IAsyncResult result = request.BeginGetResponse(
new AsyncCallback(UpdateItem),state);

semantics. This will speed up your operation by an order of magnitude since
it offers parallelism (multiple requests in operation at the same time).
Peter

--
Co-founder, Eggheadcafe.com developer portal:
http://www.eggheadcafe.com
UnBlog:
http://petesbloggerama.blogspot.com




Jim Toth said:
What I meant by "MultithreadedMethod" is that the method is called many times
on many different threads.

I've seen the asynchronous calls for GetRequestStream... will they get me
around potential contention issues mentioned?

Peter Bromberg said:
Aside from the other comments, the title of your method "MultithreadedMethod"
is confusing, as it isn't multithreaded at all the way it is written.
If you really have a lot of this going on, you probably want to look at the
asynchronous BeginXXX / EndXXX overloads which use the Threadpool "under the
hood" - or use a custom threadpool.
Peter

--
Co-founder, Eggheadcafe.com developer portal:
http://www.eggheadcafe.com
UnBlog:
http://petesbloggerama.blogspot.com




Jim Toth said:
I have a multithreaded application that is posting XML requests to a remote
web site. As more and more calls are made to
HttpWebRequest.GetRequestStream, the length of time increases for each call
from an initial 2-3 seconds to anywhere from 15-30 seconds. What am I doing
that is causing this method call's duration to increase and how do I make it
run faster? Below is some pseudo-code for the method that is called in a
multi-threaded fashion...

- Jim

private void MultithreadedMethod()
{
string url = ConfigurationManager.AppSettings["providerUrl"];

request = (HttpWebRequest)WebRequest.Create(url);
request.Method = "POST";
request.ContentType = "text/xml";

startCount = Environment.TickCount;
using (Stream s = request.GetRequestStream())
{
elapsedCount = Environment.TickCount - startCount;
//the elapsed time increases with each call
EventLog.WriteEntry("request.GetRequestStream() = " + elapsedCount,
EventLogEntryType.Information);

using (StreamWriter writer = new StreamWriter(s))
{
writer.WriteLine(requestXml);
writer.Flush();
writer.Close();
}

s.Close();
}


request.Timeout = requestTimeoutMilliseconds;
response = request.GetResponse();

//handle the response here
}
Click to expand...
Click to expand...
Click to expand...
 
J

John Murray

If you are using the 1.1 or earlier version of the Framework, never make
calls on WebClient or WebRequest from within the threadpool -- as the
calls on those objects are also implemented using the threadpool (even
if you are calling them synchronously) ...... the result is a very high
potential for a deadlock.
 
G

Guest

It looks like I'm running into some issues with
ServicePointManager.DefaultConnectionLimit, which is defaulted to 2, because
I'm sending data to the same web site URL over and over. I tried bumping
this up to 10, 15, and 20 and had to bump up the number of threads in the
ThreadPool as a result, however I'm seeing only marginal gains in
performance. Is there a configuration setting, either for .NET or Windows
Server 2003, that I should be looking at, or am I missing something else?

Peter Bromberg said:
Correct, which is a misnomer. If it is called many different times, you
should probably be using the Threadpool and the asynchronous
IAsyncResult result = request.BeginGetResponse(
new AsyncCallback(UpdateItem),state);

semantics. This will speed up your operation by an order of magnitude since
it offers parallelism (multiple requests in operation at the same time).
Peter

--
Co-founder, Eggheadcafe.com developer portal:
http://www.eggheadcafe.com
UnBlog:
http://petesbloggerama.blogspot.com




Jim Toth said:
What I meant by "MultithreadedMethod" is that the method is called many times
on many different threads.

I've seen the asynchronous calls for GetRequestStream... will they get me
around potential contention issues mentioned?

Peter Bromberg said:
Aside from the other comments, the title of your method "MultithreadedMethod"
is confusing, as it isn't multithreaded at all the way it is written.
If you really have a lot of this going on, you probably want to look at the
asynchronous BeginXXX / EndXXX overloads which use the Threadpool "under the
hood" - or use a custom threadpool.
Peter

--
Co-founder, Eggheadcafe.com developer portal:
http://www.eggheadcafe.com
UnBlog:
http://petesbloggerama.blogspot.com




:

I have a multithreaded application that is posting XML requests to a remote
web site. As more and more calls are made to
HttpWebRequest.GetRequestStream, the length of time increases for each call
from an initial 2-3 seconds to anywhere from 15-30 seconds. What am I doing
that is causing this method call's duration to increase and how do I make it
run faster? Below is some pseudo-code for the method that is called in a
multi-threaded fashion...

- Jim

private void MultithreadedMethod()
{
string url = ConfigurationManager.AppSettings["providerUrl"];

request = (HttpWebRequest)WebRequest.Create(url);
request.Method = "POST";
request.ContentType = "text/xml";

startCount = Environment.TickCount;
using (Stream s = request.GetRequestStream())
{
elapsedCount = Environment.TickCount - startCount;
//the elapsed time increases with each call
EventLog.WriteEntry("request.GetRequestStream() = " + elapsedCount,
EventLogEntryType.Information);

using (StreamWriter writer = new StreamWriter(s))
{
writer.WriteLine(requestXml);
writer.Flush();
writer.Close();
}

s.Close();
}


request.Timeout = requestTimeoutMilliseconds;
response = request.GetResponse();

//handle the response here
}
Click to expand...
Click to expand...
Click to expand...
 
G

Guest

make sure you close the Response object along with any streams /
streamreaders each time. Aside from that, I've seen a lot of old newsgroups
threads on upping the DefaultConnectionLimit with mixed results. It could be
at the server too, you know.
Peter
 
Top