Internal Server Error, WebRequest.GetResponse

[mfreeman]

I have a VB.NET 2005 Windows application that worked fine when I ran it
a month ago, and now it is throwing an exception ("The remote server
returned an error: (500) Internal Server Error.") and I don't know why.

The application screen scrapes a web page. If I paste the URL in my
browser's address bar and the expected page comes up just fine.

When I went through the hassle of getting the real response from within
the exception object, it is just the standard ASP.NET error page:

"A first chance exception of type 'System.Net.WebException' occurred in
System.dll
Server Error in '/Fundnet' Application.
Runtime Error
Description: An application error occurred on the server. The current
custom error settings for this application prevent the details of the
application error from being viewed remotely (for security reasons). It
could, however, be viewed by browsers running on the local server
machine. "

My code is quite simple, and the exception is thrown by the last of
these 3 lines:
Dim strURL As String =
"http://quicktake.morningstar.com/fundnet/PrintReport.aspx?intCountThisReadry=USA&Symbol=GATEX"
Dim myRequest As WebRequest = WebRequest.Create(strURL)
Dim myResponse As WebResponse = myRequest.GetResponse()

Does anyone have any idea what could cause this?

(The only thing I can think of that I've changed on my end recently is
that I installed FireFox 2.0 and made it my default browser.
Hopefully, that is irrelevant but I mention it just in case.)

I thoroughly searched Google and Google Groups before posting this and
have been unable to find a relevant article. Thanks for any guidance
you can provide.

- Mark Freeman

 

[mfreeman]

I used a monitor to see the actual HTTP traffic and found that there
were other headers being sent by the browser when requesting the URL,
so I added those items to the header in my .NET code, and now I can get
the page to load correctly -- sometimes.

Dim myRequest As HttpWebRequest = CType(WebRequest.Create(strURL),
HttpWebRequest)
myRequest.CookieContainer = New CookieContainer()
myRequest.CookieContainer.SetCookies(myRequest.RequestUri,
"fp=020114125810442111")
myRequest.ContentType =
"text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5"
myRequest.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US;
rv:1.8.1) Gecko/20061010 Firefox/2.0"
myRequest.Headers.Add(HttpRequestHeader.AcceptLanguage,
"en-us,en;q=0.5")
myRequest.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip,
deflate")
myRequest.Headers.Add(HttpRequestHeader.AcceptCharset,
"ISO-8859-1,utf-8;q=0.7,*;q=0.7")
myRequest.Headers.Add(HttpRequestHeader.Pragma, "nocache")

I do this in a loop, using a different parameter in the URL (after the
question mark at the end). Intermittently, it fails with the 500
error. I looked at the monitor again and found something strange.

On some attempts, the request goes out on the wire like this (and
succeeds):

GET /fundnet/PrintReport.aspx?Symbol=AABPX HTTP/1.1
Content-Type:
text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1)
Gecko/20061010 Firefox/2.0
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip, deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Pragma: nocache
Host: quicktake.morningstar.com
Cookie: fp=020114125810442111
Proxy-Connection: Keep-Alive

On other attempts, the request looks like this (and gets the
exception):

GET
http://quicktake.morningstar.com/fundnet/PrintReport.aspx?Symbol=AABPX
HTTP/1.1
Content-Type:
text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1)
Gecko/20061010 Firefox/2.0
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip, deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Pragma: nocache
Host: quicktake.morningstar.com
Cookie: fp=020114125810442111

Note that the one that fails has the full URL in the GET (absolute),
whereas the one that succeeds does not include the "http://" prefix or
the server in the GET (relative). Also, the successful one has
"Proxy-Connection: Keep-Alive" and the one that fails does not. I am
not doing anything differently between calls, so it appears to be
something weird within .NET.

Does anyone have any ideas?

- Mark