allowing http redirects??

  • Thread starter Thread starter Guest
  • Start date Start date
G

Guest

I am trying to crawl the web using the following code snippet.

HttpWebRequest req = (HttpWebRequest)WebRequest.Create(url);
req.UserAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Win32)";

....


By default, the req.AllowAutoRedirect is true and
MaximumAutomaticRedirections is 50.


When I try to crawl the following URL.

http://citeseer.ist.psu.edu/rd/5581...cs343zSzpszSzpathprof.pdf/ball96efficient.pdf

I get NameResolutionFailure exception. However, I am able to open this URL
from the browser and it gets redirected to:

http://citeseer.ist.psu.edu/cache/p...cs343zSzpszSzpathprof.pdf/ball96efficient.pdf


How do I force my C# code to go to the redirected url?
 
Jason said:
I am trying to crawl the web using the following code snippet.

HttpWebRequest req = (HttpWebRequest)WebRequest.Create(url);
req.UserAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Win32)";

...


By default, the req.AllowAutoRedirect is true and
MaximumAutomaticRedirections is 50.


When I try to crawl the following URL.

http://citeseer.ist.psu.edu/rd/55811103,653,1,0.25,Download/http://cit
eseer.ist.psu.edu/cache/papers/cs/7145/http:zSzzSzwww.stanford.eduzSzc
lasszSzcs343zSzpszSzpathprof.pdf/ball96efficient.pdf

I get NameResolutionFailure exception. However, I am able to open
this URL from the browser and it gets redirected to:

http://citeseer.ist.psu.edu/cache/papers/cs/7145/http:zSzzSzwww.stanfo
rd.eduzSzclasszSzcs343zSzpszSzpathprof.pdf/ball96efficient.pdf


How do I force my C# code to go to the redirected url?
Click to expand...

What .NET version do you use? I can get this document woth no problem:

HTTP/1.1 200 OK
Date: Wed, 01 Jun 2005 20:52:37 GMT
Server: Apache/2.0.53 (Unix)
Last-Modified: Wed, 08 Nov 2000 19:17:28 GMT
ETag: "17801be-21d36-c5214200"
Accept-Ranges: bytes
Content-Length: 138550
Connection: close
Content-Type: application/pdf

Cheers,
 
Joerg

My .NET version is 2.0.4.

I assume it worked for you from C# code. Did you set any property in
HttpWebRequest to make it work?

My code can open the redirected URL, but not the original url (with rd in it).

Jason
 
Jason said:
Joerg

My .NET version is 2.0.4.

I assume it worked for you from C# code. Did you set any property in
HttpWebRequest to make it work?
Click to expand...

Nothing special, but I tried it using .NET 1.1 SP1. It doesn't work for
me in .NET 2.0 Beta 2 either.


Cheers,
 
Back
Top