Automate screen scraping: How to programmically "push" a Login button on another web page?

[shmulik28]

I'm hoping to write a utility program that will navigate to another web
site, logon with my username/password, and download data automatically
on a schedule.

When I use HttpWebRequest to get the initial logon screen, there are
multiple embedded forms (Login, Forgot your Password, etc).

I want to be able to pass my username/password information to the web
page's login form and "push" the button (on their webpage, this is done
through some javascript code). When I view the source code of this
page, it appears that several hidden variables are set, so I imagine
the I would need to get/accept/return cookies.

I'm not sure how to "direct" the infomration to a page's particular
"form" - in my case, "pushing" the Login button.

Is there an example to automating this type of login that someone could
share with me?

Thanks!

 

[Brock Allen]

You'll have to use HttpWebRequest to do a HTTP POST passing values for the
IDs of the username and password fields. I'd suggest downloading Fiddler
to see the HTTP traffic and it'll make what's being passed over the HTTP
protocol seem so much more clear.

http://www.fiddlertool.com/fiddler/

-Brock
DevelopMentor
http://staff.develop.com/ballen

 

[shmulik28]

Brock thanks - your utility looks like a great help!.

One last question if I may - when I try and do an HttpWebRequest to the
main logon page, I get back a page with this error message:

"Please enable Javascript in your browser to use [the website]"

I've set the UserAgent to:

...
request.UserAgent = "Mozilla/4.0";

Hoping that it just needed to think it had a compatible browser ...
what do I need set in order to let the page think that javascript is
enabled?

Thanks!

 

[Brock Allen]

Brock thanks - your utility looks like a great help!.

It's not mine -- It's Microsoft's

"Please enable Javascript in your browser to use [the website]"

I've set the UserAgent to:

...
request.UserAgent = "Mozilla/4.0";
Hoping that it just needed to think it had a compatible browser ...
what do I need set in order to let the page think that javascript is
enabled?

I'd try one of these until it works (I'm sure you can guess the ones to try
first):

http://www.zytrax.com/tech/web/browser_ids.htm

-Brock
DevelopMentor
http://staff.develop.com/ballen