a simple crawler

[GeezerButler]

Hi, I am trying to write a simple crawler. I have no previous
experience in bots so beware of stupid questions

The url i want to connect to is "http://someSiteName/landing.aspx?
prg=ABC&prg=DEF&prg=GHI"
When i type this in a browser, the server redirects to some other page
(http://someSiteName/otherPage.aspx) and the page gets opened but when
i try to connect by the WebRequest class, i am able to get the
response but it contains some custom error message of that site.

So my question is that what is it that the browser can handle but the
WebRequest class cant?

 

[Kevin Spencer]

Hard to say for sure, but some sites block requests that don't have a
UserAgent header, for example.

--
HTH,

Kevin Spencer
Chicken Salad Surgeon
Microsoft MVP

 

[london calling]

Check out HttpWebRequest.AllowAutoRedirect in the docs

 

[london calling]

It is also possible that the web browser is being redirected via javascript
rather than Http Headers..

 

[GeezerButler]

Thanks folks but i wrote a windows app with a
System.Windows.Forms.WebBrowser class which made this a whole lot
easier than the console app i was trying to write

 

[Ravi Bhavnani]

Cool!

Also see these 2 articles by me:

WebResourceProvider goes .NET
http://www.codeproject.com/KB/IP/WebResourceProviderDotNet.aspx

Domain Walker
http://www.codeproject.com/KB/IP/DomainWalkerDotNet.aspx

/ravi