Screenscraping question ... URGENT

[Bit Byte]

I need to download some publicly available data from a website. I can
access the data manually, without any problem when I use my browser,
however, I am trying to automate the donloading of the data, using a C#
application.

However when I try to use an "HTTP get" from C#, I get a message from
the server saying that "No Automatic downloads allowed".

I have two questions:

1). Has someone come accross this problem before?
2). How do I get round this ? - some guidelines, pseudocode or even a
link to some sample code would be greatly appreciated...

Additionally, I would be grateful if someone could explain technically,
whats going on.

Misc info: The site I'm tring to access is running IIS on .Net 1.1 framework

 

[Greg Young]

Rather obviously it would not be ethical for us to give you a way to bypass
someone's system when they obviously do not want you to automatically
download and screen scrape from their site.

I will however point you in the right direction ... try using a tool like
http://www.securitysupervisor.com/s/http sniffer.php and see what is
different between the http posts.

Cheers,

Greg Young
MVP - C#
http://codebetter.com/blogs/gregyoung

 

[Robbe Morris [C# MVP]]

My guess is that if they've gone that far, they've probably
gone even further. You are probably wasting your time.

 

[Joerg Jooss]

Thus wrote Bit,

I need to download some publicly available data from a website. I can
access the data manually, without any problem when I use my browser,
however, I am trying to automate the donloading of the data, using a
C# application.

However when I try to use an "HTTP get" from C#, I get a message from
the server saying that "No Automatic downloads allowed".

I have two questions:

1). Has someone come accross this problem before?
2). How do I get round this ? - some guidelines, pseudocode or even a
link to some sample code would be greatly appreciated...
Additionally, I would be grateful if someone could explain
technically, whats going on.

Try setting the HttpWebRequest.UserAgent property to a known browser, such
as
"Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.8.0.4) Gecko/20060508
Firefox/1.5.0.4".

Cheers,

 

[Jon Skeet [C# MVP]]

I need to download some publicly available data from a website. I can
access the data manually, without any problem when I use my browser,
however, I am trying to automate the donloading of the data, using a C#
application.

However when I try to use an "HTTP get" from C#, I get a message from
the server saying that "No Automatic downloads allowed".

That should suggest to you that they really don't want you to download
the data automatically. If you feel you have a valid reason for doing
so, why not email them? Otherwise, show some respect for their wishes.

Jon

 

[Bit Byte]

I need to download some publicly available data from a website. I can
access the data manually, without any problem when I use my browser,
however, I am trying to automate the donloading of the data, using a C#
application.

However when I try to use an "HTTP get" from C#, I get a message from
the server saying that "No Automatic downloads allowed".

I have two questions:

1). Has someone come accross this problem before?
2). How do I get round this ? - some guidelines, pseudocode or even a
link to some sample code would be greatly appreciated...

Additionally, I would be grateful if someone could explain technically,
whats going on.

Misc info: The site I'm tring to access is running IIS on .Net 1.1
framework

I can understand the "moral" objections .... but as I said, this data is
already freely available on the internet. I normally manually download
it myself, but I am going away on holiday for 2 weeks, and I will not
have access to a computer, (and MO, I don't want/have anybody o ask to
do this chore for me whilst I'm away), so all I want to do, is to leave
my computer switched on, and let my program download the data on my
behalf - its not like I will be using a mega computer to overwhelm the
server or anything (I can understand why the site may want to prevent
activities like that may quickly overwhelm the server), but I want to be
able to do what I normally do everyday, using my browser - thats all.