Is it possible to screen scrape a secure site (HTTPS).....

[Ollie]

I know you can screen scrape a website using the System.Net.HttpWebResponse
& System.Net.HttpWebRequest classes.

But how do you screen scrape a secured website (https) that takes a username
& password, I guess what I am asking where does the username & password
go and where do you store any returned token for further requests...

Cheers

Ollie

 

[Max]

This might work?
https://username:[email protected]

 

[Eric Lawrence [MSFT]]

There's probably a property on the object to allow you to add your
credentials.

If not, it's still pretty easy. First, figure out what protocol the secure
site is using for authentication-- is it BASIC, DIGEST, Negotiate, or
whatever. If it's basic, it's simple-- just add an Authorization header
with the base-64 encoded usernameassword string. If it's digest or
Negotiate, it gets trickier.

--
Thanks,

Eric Lawrence
Program Manager
Assistance and Worldwide Services

This posting is provided "AS IS" with no warranties, and confers no rights.

 

[Chad Z. Hower aka Kudzu]

If not, it's still pretty easy. First, figure out what protocol the secure
site is using for authentication-- is it BASIC, DIGEST, Negotiate, or
whatever. If it's basic, it's simple-- just add an Authorization header
with the base-64 encoded usernameassword string. If it's digest or
Negotiate, it gets trickier.

Indy supports all of these authentications and more. Its also free.

http://www.indyproject.org/

The only problem is that the .net version doesnt support SSL yet. BUT you
could bundle it with an SSL tunnel if your options are limited.