Simulating a Browser?

A

Arne Vajhøj

Is this commonly done? I'm thinking it would be extremely useful.

I think it is relative common to send HTTP requests from C# code
(WebClient or HttpWebRequest). Sometimes it fake headers to completely
look like a browser.

It is also possible to embed a web browser in a web form.

Arne
 
D

Davej

I think it is relative common to send HTTP requests from C#
code (WebClient or HttpWebRequest). Sometimes it fake
headers to completely look like a browser.

It is also possible to embed a web browser in a web form.

Arne

Well, I'm interested in the very simple case of communicating with a
website using a barebones page it has set aside for this automated
purpose. Is that sort of thing pretty trivial to get working? Thanks.
 
A

Arne Vajhøj

Well, I'm interested in the very simple case of communicating with a
website using a barebones page it has set aside for this automated
purpose. Is that sort of thing pretty trivial to get working? Thanks.

If you can use WebClient then it is rather trivial.

WebClient wc = new WebClient();
string html = wc.DownloadString(url);

Arne
 
D

Davej

If you can use WebClient then it is rather trivial.

WebClient wc = new WebClient();
string html = wc.DownloadString(url);

Arne

So far I can see the file at the URL, but I can't get a response to a
post. The UploadString() example looks a bit too simple. Shouldn't the
data string have a format more like "name1=value1&name2=value2" ? Or
do I have to do some other setup stuff?

http://msdn.microsoft.com/en-us/library/d0d3595k(v=vs.110).aspx

Thanks.
 
A

Arne Vajhøj

So far I can see the file at the URL, but I can't get a response to a
post. The UploadString() example looks a bit too simple. Shouldn't the
data string have a format more like "name1=value1&name2=value2" ? Or
do I have to do some other setup stuff?

http://msdn.microsoft.com/en-us/library/d0d3595k(v=vs.110).aspx

It is probably UploadString you need to use and you can send data
with that.

But if you want more control, then look at HttpWebRequest
(I can find an example if needed).

Arne
 
D

Davej

It is probably UploadString you need to use and you can send data
with that.

But if you want more control, then look at HttpWebRequest
(I can find an example if needed).

Arne

I looked up HttpWebRequest but the constructor is marked "obsolete."
See...

http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest(v=vs.110).aspx

I tried UploadString(), UploadData() and UploadValues() but the page
never seems to see the POST. It just responds with what looks like an
initial rendering.

Here is the UploadValues() code...

public String UploadValuesTest() {

WebClient client = new WebClient();

//Console.Write("\nPlease enter the URI to post data to :
");
string uriString = @"http://www.stlnetwork.net/
default.aspx";

// Create a new NameValueCollection instance to hold some
custom parameters to be posted to the URL.
NameValueCollection NameValPairs = new
NameValueCollection();

//<form name="form1" method="post" action="default.aspx"
id="form1">
//<input name="txtUserID" type="text" id="txtUserID" />
//<input name="txtPassword" type="password"
id="txtPassword" />
//<input type="submit" name="btnLogin" value="Login"
id="btnLogin" />

// Add necessary parameter/value pairs to the name/value
container.
//"txtUserID=Userabc txtPassword=Passwd1";
NameValPairs.Add("txtUserID", "Userabc");
NameValPairs.Add("txtPassword", "Passwd1");

// 'The Upload(String,NameValueCollection)' implicitly
method sets HTTP POST as the request method.
byte[] responseArray = client.UploadValues(uriString,
NameValPairs);

// Decode and display the response.
return Encoding.ASCII.GetString(responseArray);

}//end
 
A

Arne Vajhøj

I looked up HttpWebRequest but the constructor is marked "obsolete."
See...

http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest(v=vs.110).aspx

You use WebRequest.Create with a HTTP URL to create a HttpWebRequest.
I tried UploadString(), UploadData() and UploadValues() but the page
never seems to see the POST. It just responds with what looks like an
initial rendering.

You need to be sure that you are POST'ing to the action URL
not the form URL.

Arne
 
A

Arne Vajhøj

You use WebRequest.Create with a HTTP URL to create a HttpWebRequest.


You need to be sure that you are POST'ing to the action URL
not the form URL.

Ooops.

I just checked your comment.

//<form name="form1" method="post" action="default.aspx" id="form1">
//<input name="txtUserID" type="text" id="txtUserID" />
//<input name="txtPassword" type="password" id="txtPassword" />
//<input type="submit" name="btnLogin" value="Login" id="btnLogin" />

If that is an ASP.NET web forms page, then it becomes a bit
tricky.

You need to send the viewstate in your POST for things
to work properly.

Look at the form source and see the hidden field.

Arne
 
D

Davej

Ooops.

I just checked your comment.

//<form name="form1" method="post" action="default.aspx" id="form1">
//<input name="txtUserID" type="text" id="txtUserID" />
//<input name="txtPassword" type="password" id="txtPassword" />
//<input type="submit" name="btnLogin" value="Login" id="btnLogin" />

If that is an ASP.NET web forms page, then it becomes a bit
tricky.

You need to send the viewstate in your POST for things
to work properly.

Look at the form source and see the hidden field.

Arne

Oh! I forgot about that hidden field! So I need to downloadData() and
then send the name/value pair of the viewstate back with the other
pairs?
 
A

Arne Vajhøj

Oh! I forgot about that hidden field! So I need to downloadData() and
then send the name/value pair of the viewstate back with the other
pairs?

Yes.

Arne
 
D

Davej

Bingo. That was it. Much thanks! Should have been obvious to me but I
just wasn't thinking to look for hidden fields.

I have some working test code now. The routines necessary to extract
the two keys in the hidden fields were rather bothersome. Now I need
to figure out how to implement the "non blocking" methods because this
process is pretty slow.
 
A

Arne Vajhøj

I have some working test code now. The routines necessary to extract
the two keys in the hidden fields were rather bothersome.

Yes - calling web forms pages programmatically is just cumbersome.
Now I need
to figure out how to implement the "non blocking" methods because this
process is pretty slow.

Well - you can either use threads/threadpool or go for the asynch
methods.

Arne
 
D

Davej

Yes - calling web forms pages programmatically is just cumbersome.


Well - you can either use threads/threadpool or go for the asynch
methods.

Arne

Are there other better schemes that I should be investigating? I have
read a little about web services, but they seem to be all about
exchanging XML data. Compared to that the various WebClient methods
seem pretty powerful. I want to be able to upload and download jpg
files and some text records. It looks like the WebClient methods will
cover that. Thanks.
 
A

Arne Vajhøj

Are there other better schemes that I should be investigating?

I don't think so.

For HTML parsing you could look at HTML Agility Pack.
I have
read a little about web services, but they seem to be all about
exchanging XML data.

Or at least a format intended for an application and not a browser.
Compared to that the various WebClient methods
seem pretty powerful. I want to be able to upload and download jpg
files and some text records. It looks like the WebClient methods will
cover that.

And if WebClient is not flexible enough then you can fallback to
HttpWebRequest.

Arne
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Top