WebResponse Class Problems

[Brent]

I'm having odd problems with the WebResponse class. Some servers are
speedy, while others don't play along at all. Consider the following pages*:

http://planetbrent.com/test.aspx?url=http://www.yahoo.com
http://planetbrent.com/test.aspx?url=http://www.msn.com
http://planetbrent.com/test.aspx?url=http://www.sec.gov
http://planetbrent.com/test.aspx?ur...5/000090266405001703/0000902664-05-001703.txt

The first two return HTML all-but instantly. The third and fourth links
-- both on the same server, I'm guessing -- take up to a minute, which
you can see if you test them. The four pages are roughly the same size
in bytes.

I'm thinking there may be an issue with the way the response comes back
from the server. Is it possible that some servers deliver data that's
confusing to the Http/WebResponse class? Or could the problem relate to
the headers I'm sending in my request.

I'd sure appreciate any pointers on handling this situation!

--Brent

*Full source code for the page.
=========================================================
<%@ Page Language="C#" %>
<%@ Import Namespace="System" %>
<%@ Import Namespace="System.Web" %>
<%@ Import Namespace="System.Net" %>
<%@ Import Namespace="System.IO" %>
<%@ Import Namespace="System.Text" %>
<%@ Import Namespace="System.Data" %>

<script language="C#" runat="server">

public void Page_Load(Object sender, EventArgs e)
{
string url = Request.QueryString["url"] != null ?
Request.QueryString["url"] : "http://www.yahoo.com";
ctrlSF.Text = "Started: " +
System.DateTime.Now.ToString("HH:mm:ss") + " | ";
ctrlText.Text = getHeader(url);
ctrlSF.Text += "Finished: " + System.DateTime.Now.ToString("HH:mm:ss");
}


public string getHeader (string strURL)
{
StringBuilder returnString = new StringBuilder();

String strreturn;
String thisRow = "";

WebResponse objResponse;
WebRequest objRequest = HttpWebRequest.Create(strURL);
objResponse = objRequest.GetResponse();

using (StreamReader sr = new
StreamReader(objResponse.GetResponseStream()))
{
while ((thisRow = sr.ReadLine()) != null)
{
returnString.Append(thisRow);
}
}
return returnString.ToString();
}

</script>

<asp:Literal id="ctrlSF" runat="server" />
<asp:Literal id="ctrlText" runat = "server" />

 

[Alvin Bruney - ASP.NET MVP]

You also have an internal limit of 2 requests per connection that may be
clogging up the lines. You can test this difference by switching the order
of the requests. Slow times for the originally fast sites would indicate a
problem related to the request/response and not the site. See if that helps
any.

--
Regards,
Alvin Bruney [MVP ASP.NET]

[Shameless Author plug]
The Microsoft Office Web Components Black Book with .NET
Now Available @ www.lulu.com/owc
Forth-coming VSTO.NET - Wrox/Wiley 2006
-------------------------------------------------------

I'm having odd problems with the WebResponse class. Some servers are
speedy, while others don't play along at all. Consider the following pages*:

http://planetbrent.com/test.aspx?url=http://www.yahoo.com
http://planetbrent.com/test.aspx?url=http://www.msn.com
http://planetbrent.com/test.aspx?url=http://www.sec.gov

http://planetbrent.com/test.aspx?ur...5/000090266405001703/0000902664-05-001703.txt

The first two return HTML all-but instantly. The third and fourth links
-- both on the same server, I'm guessing -- take up to a minute, which
you can see if you test them. The four pages are roughly the same size
in bytes.

I'm thinking there may be an issue with the way the response comes back
from the server. Is it possible that some servers deliver data that's
confusing to the Http/WebResponse class? Or could the problem relate to
the headers I'm sending in my request.

I'd sure appreciate any pointers on handling this situation!

--Brent

*Full source code for the page.
=========================================================
<%@ Page Language="C#" %>
<%@ Import Namespace="System" %>
<%@ Import Namespace="System.Web" %>
<%@ Import Namespace="System.Net" %>
<%@ Import Namespace="System.IO" %>
<%@ Import Namespace="System.Text" %>
<%@ Import Namespace="System.Data" %>

<script language="C#" runat="server">

public void Page_Load(Object sender, EventArgs e)
{
string url = Request.QueryString["url"] != null ?
Request.QueryString["url"] : "http://www.yahoo.com";
ctrlSF.Text = "Started: " +
System.DateTime.Now.ToString("HH:mm:ss") + " | ";
ctrlText.Text = getHeader(url);
ctrlSF.Text += "Finished: " + System.DateTime.Now.ToString("HH:mm:ss");
}


public string getHeader (string strURL)
{
StringBuilder returnString = new StringBuilder();

String strreturn;
String thisRow = "";

WebResponse objResponse;
WebRequest objRequest = HttpWebRequest.Create(strURL);
objResponse = objRequest.GetResponse();

using (StreamReader sr = new
StreamReader(objResponse.GetResponseStream()))
{
while ((thisRow = sr.ReadLine()) != null)
{
returnString.Append(thisRow);
}
}
return returnString.ToString();
}

</script>

<asp:Literal id="ctrlSF" runat="server" />
<asp:Literal id="ctrlText" runat = "server" />

 

[John Timney \( MVP \)]

It could easily be as simple as the first two links are cached or the web
servers are slow/traffic is high, therefore delivered very quickly or very
slowly as a response from the web server.

You need to do some testing first to determine if the pages respond as fast
or as slow as each other without the .net classes in the loop - and build
your findings out of that.

--
Regards

John Timney
ASP.NET MVP
Microsoft Regional Director

I'm having odd problems with the WebResponse class. Some servers are
speedy, while others don't play along at all. Consider the following
pages*:

http://planetbrent.com/test.aspx?url=http://www.yahoo.com
http://planetbrent.com/test.aspx?url=http://www.msn.com
http://planetbrent.com/test.aspx?url=http://www.sec.gov
http://planetbrent.com/test.aspx?ur...5/000090266405001703/0000902664-05-001703.txt

The first two return HTML all-but instantly. The third and fourth links --
both on the same server, I'm guessing -- take up to a minute, which you
can see if you test them. The four pages are roughly the same size in
bytes.

I'm thinking there may be an issue with the way the response comes back
from the server. Is it possible that some servers deliver data that's
confusing to the Http/WebResponse class? Or could the problem relate to
the headers I'm sending in my request.

I'd sure appreciate any pointers on handling this situation!

--Brent

*Full source code for the page.
=========================================================
<%@ Page Language="C#" %>
<%@ Import Namespace="System" %>
<%@ Import Namespace="System.Web" %>
<%@ Import Namespace="System.Net" %>
<%@ Import Namespace="System.IO" %>
<%@ Import Namespace="System.Text" %>
<%@ Import Namespace="System.Data" %>

<script language="C#" runat="server">

public void Page_Load(Object sender, EventArgs e)
{
string url = Request.QueryString["url"] != null ?
Request.QueryString["url"] : "http://www.yahoo.com";
ctrlSF.Text = "Started: " + System.DateTime.Now.ToString("HH:mm:ss") +
" | ";
ctrlText.Text = getHeader(url);
ctrlSF.Text += "Finished: " +
System.DateTime.Now.ToString("HH:mm:ss");
}


public string getHeader (string strURL)
{
StringBuilder returnString = new StringBuilder();

String strreturn;
String thisRow = "";

WebResponse objResponse;
WebRequest objRequest = HttpWebRequest.Create(strURL);
objResponse = objRequest.GetResponse();

using (StreamReader sr = new
StreamReader(objResponse.GetResponseStream()))
{
while ((thisRow = sr.ReadLine()) != null)
{
returnString.Append(thisRow);
}
}
return returnString.ToString();
}

</script>

<asp:Literal id="ctrlSF" runat="server" />
<asp:Literal id="ctrlText" runat = "server" />

 

[Brent]

Thanks for your responses!

Alvin: I know about the 2 requests per connection issue. Actually, I'm
not requesting anything sequentially here, I'm requesting one site and
then seeing how long it takes to come back. I then close the
WebResponse, reopen it, and then request the next site. I'm not sure how
the connection requests would be affecting the speed.

John: I'm curious how the first two links could be cached, while the
others are not. I've accessed all pages in various ways, both with the
code and directly from a browser on the same machine, and it's only the
code that slows down on http://www.sec.gov. How would I test the
external pages "without the .net classes in the loop" other than looking
at them in a browser?

--Brent

 

[John Timney \( MVP \)]

I mean they could be getting delivered via a cache server at the delivery
end, and hence are being delivered somewhat faster. If they appear slow
only in .net and not in your browser this is not likely to be the case.

Have you tried only requesting the second two via your code, or trying an
online analyzer?

http://www.websiteoptimization.com/services/analyze/

--
Regards

John Timney
ASP.NET MVP
Microsoft Regional Director

 

[brentbackup]

Hmmm...well, I think I've tried every thing I can think of to make this
code work quickly. The odd thing is that two weeks ago (or so), it
worked just fine. I keep thinking, perhaps, that the server is doing
some tarpitting. Is it possible that an Apache server could slow down a
"non-browser" request? Could I mimic a browser in code? Are there any
possible conflicts in the way the data gets returned or interpreted by
my server -- such that my server gets confused and slows down? Why is
the site just fine when viewed through MSIE on my server, but not
through the code? Why are all the other sites speedy, with ONLY (it
seems) http://www.sec.gov being slow?

It's really frustrating!

Thanks for your replies!

--Brent

 

[Joerg Jooss]

Hmmm...well, I think I've tried every thing I can think of to make
this code work quickly. The odd thing is that two weeks ago (or so),
it worked just fine. I keep thinking, perhaps, that the server is
doing some tarpitting. Is it possible that an Apache server could
slow down a "non-browser" request? Could I mimic a browser in code?

Yes -- simply set the UserAgent property to whatever your preferred
browser would send.

Cheers,