Is it possible to screen scrape a secure site (HTTPS).....

  • Thread starter Thread starter Ollie
  • Start date Start date
O

Ollie

I know you can screen scrape a website using the System.Net.HttpWebResponse
& System.Net.HttpWebRequest classes.

But how do you screen scrape a secured website (https) that takes a username
& password, I guess what I am asking where does the username & password
go and where do you store any returned token for further requests...

Cheers

Ollie
 
There's probably a property on the object to allow you to add your
credentials.

If not, it's still pretty easy. First, figure out what protocol the secure
site is using for authentication-- is it BASIC, DIGEST, Negotiate, or
whatever. If it's basic, it's simple-- just add an Authorization header
with the base-64 encoded username:password string. If it's digest or
Negotiate, it gets trickier.

--
Thanks,

Eric Lawrence
Program Manager
Assistance and Worldwide Services

This posting is provided "AS IS" with no warranties, and confers no rights.
 
Eric Lawrence said:
If not, it's still pretty easy. First, figure out what protocol the secure
site is using for authentication-- is it BASIC, DIGEST, Negotiate, or
whatever. If it's basic, it's simple-- just add an Authorization header
with the base-64 encoded username:password string. If it's digest or
Negotiate, it gets trickier.
Click to expand...

Indy supports all of these authentications and more. Its also free.

http://www.indyproject.org/

The only problem is that the .net version doesnt support SSL yet. BUT you
could bundle it with an SSL tunnel if your options are limited.
 
Back
Top