PC Review


Reply
Thread Tools Rating: Thread Rating: 3 votes, 3.67 average.

Simulating a Browser?

 
 
Arne Vajhøj
Guest
Posts: n/a
 
      12th Apr 2012
On 4/11/2012 7:22 PM, Davej wrote:
> On Apr 11, 6:09 pm, Arne Vajhøj<a...@vajhoej.dk> wrote:
>> On 4/11/2012 7:06 PM, Arne Vajhøj wrote:
>>> On 4/11/2012 7:01 PM, Davej wrote:
>>>> On Apr 11, 4:41 pm, Arne Vajhøj<a...@vajhoej.dk> wrote:
>>>>> On 4/11/2012 12:48 PM, Davej wrote:
>>>>>> On Apr 7, 8:29 am, Arne Vajhøj<a...@vajhoej.dk> wrote:
>>>>>>> On 4/7/2012 9:18 AM, Davej wrote:
>>>>>>>> On Apr 7, 8:04 am, Arne Vajhøj<a...@vajhoej.dk> wrote:
>>>>>>>>> On 4/7/2012 8:26 AM, Davej wrote:

>>
>>>>>>>>>> Is this commonly done? I'm thinking it would be extremely useful.

>>
>>>>>>>>> I think it is relative common to send HTTP requests from C#
>>>>>>>>> code (WebClient or HttpWebRequest). Sometimes it fake
>>>>>>>>> headers to completely look like a browser.

>>
>>>>>>>>> It is also possible to embed a web browser in a web form.

>>
>>>>>>>> Well, I'm interested in the very simple case of communicating with a
>>>>>>>> website using a barebones page it has set aside for this automated
>>>>>>>> purpose. Is that sort of thing pretty trivial to get working? Thanks.

>>
>>>>>>> If you can use WebClient then it is rather trivial.

>>
>>>>>>> WebClient wc = new WebClient();

>>
>>>>>> So far I can see the file at the URL, but I can't get a response to a
>>>>>> post. The UploadString() example looks a bit too simple. Shouldn't the
>>>>>> data string have a format more like "name1=value1&name2=value2" ? Or
>>>>>> do I have to do some other setup stuff?

>>
>>>>>> http://msdn.microsoft.com/en-us/libr...vs.110%29.aspx

>>
>>>>> It is probably UploadString you need to use and you can send data
>>>>> with that.

>>
>>>>> But if you want more control, then look at HttpWebRequest
>>>>> (I can find an example if needed).

>>
>>>> I looked up HttpWebRequest but the constructor is marked "obsolete."
>>>> See...

>>
>>>> http://msdn.microsoft.com/en-us/libr...webrequest%28v...

>>
>>> You use WebRequest.Create with a HTTP URL to create a HttpWebRequest.

>>
>>>> I tried UploadString(), UploadData() and UploadValues() but the page
>>>> never seems to see the POST. It just responds with what looks like an
>>>> initial rendering.

>>
>>> You need to be sure that you are POST'ing to the action URL
>>> not the form URL.

>>
>> Ooops.
>>
>> I just checked your comment.
>>
>> //<form name="form1" method="post" action="default.aspx" id="form1">
>> //<input name="txtUserID" type="text" id="txtUserID" />
>> //<input name="txtPassword" type="password" id="txtPassword" />
>> //<input type="submit" name="btnLogin" value="Login" id="btnLogin" />
>>
>> If that is an ASP.NET web forms page, then it becomes a bit
>> tricky.
>>
>> You need to send the viewstate in your POST for things
>> to work properly.
>>
>> Look at the form source and see the hidden field.

>
> Oh! I forgot about that hidden field! So I need to downloadData() and
> then send the name/value pair of the viewstate back with the other
> pairs?


Yes.

Arne


 
Reply With Quote
 
 
 
 
Davej
Guest
Posts: n/a
 
      12th Apr 2012
On Apr 11, 6:30*pm, Arne Vajhøj <a...@vajhoej.dk> wrote:
> On 4/11/2012 7:22 PM, Davej wrote:
> > On Apr 11, 6:09 pm, Arne Vajhøj<a...@vajhoej.dk> *wrote:
> >> On 4/11/2012 7:06 PM, Arne Vajhøj wrote:
> >>> On 4/11/2012 7:01 PM, Davej wrote:
> >>>> On Apr 11, 4:41 pm, Arne Vajhøj<a...@vajhoej.dk> *wrote:
> >>>>> On 4/11/2012 12:48 PM, Davej wrote:
> >>>>>> On Apr 7, 8:29 am, Arne Vajhøj<a...@vajhoej.dk> *wrote:
> >>>>>>> On 4/7/2012 9:18 AM, Davej wrote:
> >>>>>>>> On Apr 7, 8:04 am, Arne Vajhøj<a...@vajhoej.dk> *wrote:
> >>>>>>>>> On 4/7/2012 8:26 AM, Davej wrote:

>
> >>>>>>>>>> Is this commonly done? I'm thinking it would be extremely useful.

>
> >>>>>>>>> I think it is relative common to send HTTP requests from C#
> >>>>>>>>> code (WebClient or HttpWebRequest). Sometimes it fake
> >>>>>>>>> headers to completely look like a browser.

>
> >>>>>>>>> It is also possible to embed a web browser in a web form.

>
> >>>>>>>> Well, I'm interested in the very simple case of communicating with a
> >>>>>>>> website using a barebones page it has set aside for this automated
> >>>>>>>> purpose. Is that sort of thing pretty trivial to get working? Thanks.

>
> >>>>>>> If you can use WebClient then it is rather trivial.

>
> >>>>>>> WebClient wc = new WebClient();

>
> >>>>>> So far I can see the file at the URL, but I can't get a response to a
> >>>>>> post. The UploadString() example looks a bit too simple. Shouldn'tthe
> >>>>>> data string have a format more like "name1=value1&name2=value2" ? Or
> >>>>>> do I have to do some other setup stuff?

>
> >>>>>>http://msdn.microsoft.com/en-us/libr...vs.110%29.aspx

>
> >>>>> It is probably UploadString you need to use and you can send data
> >>>>> with that.

>
> >>>>> But if you want more control, then look at HttpWebRequest
> >>>>> (I can find an example if needed).

>
> >>>> I looked up HttpWebRequest but the constructor is marked "obsolete."
> >>>> See...

>
> >>>>http://msdn.microsoft.com/en-us/libr...webrequest%28v....

>
> >>> You use WebRequest.Create with a HTTP URL to create a HttpWebRequest.

>
> >>>> I tried UploadString(), UploadData() and UploadValues() but the page
> >>>> never seems to see the POST. It just responds with what looks like an
> >>>> initial rendering.

>
> >>> You need to be sure that you are POST'ing to the action URL
> >>> not the form URL.

>
> >> Ooops.

>
> >> I just checked your comment.

>
> >> //<form name="form1" method="post" action="default.aspx" id="form1">
> >> //<input name="txtUserID" type="text" id="txtUserID" />
> >> //<input name="txtPassword" type="password" id="txtPassword" />
> >> //<input type="submit" name="btnLogin" value="Login" id="btnLogin" />

>
> >> If that is an ASP.NET web forms page, then it becomes a bit
> >> tricky.

>
> >> You need to send the viewstate in your POST for things
> >> to work properly.

>
> >> Look at the form source and see the hidden field.

>
> > Oh! I forgot about that hidden field! So I need to downloadData() and
> > then send the name/value pair of the viewstate back with the other
> > pairs?

>
> Yes.
>
> Arne


Bingo. That was it. Much thanks! Should have been obvious to me but I
just wasn't thinking to look for hidden fields.
 
Reply With Quote
 
 
 
 
Davej
Guest
Posts: n/a
 
      12th Apr 2012
On Apr 11, 6:51*pm, Davej <galt...@hotmail.com> wrote:
> On Apr 11, 6:30*pm, Arne Vajhøj <a...@vajhoej.dk> wrote:
> > On 4/11/2012 7:22 PM, Davej wrote:
> > > On Apr 11, 6:09 pm, Arne Vajhøj<a...@vajhoej.dk> *wrote:
> > >> On 4/11/2012 7:06 PM, Arne Vajhøj wrote:
> > >>> On 4/11/2012 7:01 PM, Davej wrote:

>
> > >>>> I tried UploadString(), UploadData() and UploadValues() but the page
> > >>>> never seems to see the POST. It just responds with what looks likean
> > >>>> initial rendering.

>
> > >>> You need to be sure that you are POST'ing to the action URL
> > >>> not the form URL.

>
> > >> Ooops.

>
> > >> I just checked your comment.

>
> > >> //<form name="form1" method="post" action="default.aspx" id="form1">
> > >> //<input name="txtUserID" type="text" id="txtUserID" />
> > >> //<input name="txtPassword" type="password" id="txtPassword" />
> > >> //<input type="submit" name="btnLogin" value="Login" id="btnLogin" />

>
> > >> If that is an ASP.NET web forms page, then it becomes a bit
> > >> tricky.

>
> > >> You need to send the viewstate in your POST for things
> > >> to work properly.

>
> > >> Look at the form source and see the hidden field.

>
> > > Oh! I forgot about that hidden field! So I need to downloadData() and
> > > then send the name/value pair of the viewstate back with the other
> > > pairs?

>
> > Yes.

>
> > Arne

>
> Bingo. That was it. Much thanks! Should have been obvious to me but I
> just wasn't thinking to look for hidden fields.


I have some working test code now. The routines necessary to extract
the two keys in the hidden fields were rather bothersome. Now I need
to figure out how to implement the "non blocking" methods because this
process is pretty slow.
 
Reply With Quote
 
Arne Vajhøj
Guest
Posts: n/a
 
      12th Apr 2012
On 4/12/2012 2:30 PM, Davej wrote:
> On Apr 11, 6:51 pm, Davej<galt...@hotmail.com> wrote:
>> On Apr 11, 6:30 pm, Arne Vajhøj<a...@vajhoej.dk> wrote:
>>> On 4/11/2012 7:22 PM, Davej wrote:
>>>> On Apr 11, 6:09 pm, Arne Vajhøj<a...@vajhoej.dk> wrote:
>>>>> On 4/11/2012 7:06 PM, Arne Vajhøj wrote:
>>>>>> On 4/11/2012 7:01 PM, Davej wrote:

>>
>>>>>>> I tried UploadString(), UploadData() and UploadValues() but the page
>>>>>>> never seems to see the POST. It just responds with what looks like an
>>>>>>> initial rendering.

>>
>>>>>> You need to be sure that you are POST'ing to the action URL
>>>>>> not the form URL.

>>
>>>>> Ooops.

>>
>>>>> I just checked your comment.

>>
>>>>> //<form name="form1" method="post" action="default.aspx" id="form1">
>>>>> //<input name="txtUserID" type="text" id="txtUserID" />
>>>>> //<input name="txtPassword" type="password" id="txtPassword" />
>>>>> //<input type="submit" name="btnLogin" value="Login" id="btnLogin" />

>>
>>>>> If that is an ASP.NET web forms page, then it becomes a bit
>>>>> tricky.

>>
>>>>> You need to send the viewstate in your POST for things
>>>>> to work properly.

>>
>>>>> Look at the form source and see the hidden field.

>>
>>>> Oh! I forgot about that hidden field! So I need to downloadData() and
>>>> then send the name/value pair of the viewstate back with the other
>>>> pairs?

>>
>>> Yes.

>>
>> Bingo. That was it. Much thanks! Should have been obvious to me but I
>> just wasn't thinking to look for hidden fields.

>
> I have some working test code now. The routines necessary to extract
> the two keys in the hidden fields were rather bothersome.


Yes - calling web forms pages programmatically is just cumbersome.

> Now I need
> to figure out how to implement the "non blocking" methods because this
> process is pretty slow.


Well - you can either use threads/threadpool or go for the asynch
methods.

Arne


 
Reply With Quote
 
Davej
Guest
Posts: n/a
 
      12th Apr 2012
On Apr 12, 4:46*pm, Arne Vajhøj <a...@vajhoej.dk> wrote:
> On 4/12/2012 2:30 PM, Davej wrote:
> > On Apr 11, 6:51 pm, Davej<galt...@hotmail.com> *wrote:
> >> On Apr 11, 6:30 pm, Arne Vajhøj<a...@vajhoej.dk> *wrote:
> >>> On 4/11/2012 7:22 PM, Davej wrote:
> >>>> On Apr 11, 6:09 pm, Arne Vajhøj<a...@vajhoej.dk> * *wrote:
> >>>>> On 4/11/2012 7:06 PM, Arne Vajhøj wrote:
> >>>>>> On 4/11/2012 7:01 PM, Davej wrote:

>
> >>>>>>> I tried UploadString(), UploadData() and UploadValues() but the page
> >>>>>>> never seems to see the POST. It just responds with what looks like an
> >>>>>>> initial rendering.

>
> >>>>>> You need to be sure that you are POST'ing to the action URL
> >>>>>> not the form URL.

>
> >>>>> Ooops.

>
> >>>>> I just checked your comment.

>
> >>>>> //<form name="form1" method="post" action="default.aspx" id="form1">
> >>>>> //<input name="txtUserID" type="text" id="txtUserID" />
> >>>>> //<input name="txtPassword" type="password" id="txtPassword" />
> >>>>> //<input type="submit" name="btnLogin" value="Login" id="btnLogin" />

>
> >>>>> If that is an ASP.NET web forms page, then it becomes a bit
> >>>>> tricky.

>
> >>>>> You need to send the viewstate in your POST for things
> >>>>> to work properly.

>
> >>>>> Look at the form source and see the hidden field.

>
> >>>> Oh! I forgot about that hidden field! So I need to downloadData() and
> >>>> then send the name/value pair of the viewstate back with the other
> >>>> pairs?

>
> >>> Yes.

>
> >> Bingo. That was it. Much thanks! Should have been obvious to me but I
> >> just wasn't thinking to look for hidden fields.

>
> > I have some working test code now. The routines necessary to extract
> > the two keys in the hidden fields were rather bothersome.

>
> Yes - calling web forms pages programmatically is just cumbersome.
>
> >*Now I need to figure out how to implement the "non blocking"
> > methods because this process is pretty slow.

>
> Well - you can either use threads/threadpool or go for the asynch
> methods.
>
> Arne


Are there other better schemes that I should be investigating? I have
read a little about web services, but they seem to be all about
exchanging XML data. Compared to that the various WebClient methods
seem pretty powerful. I want to be able to upload and download jpg
files and some text records. It looks like the WebClient methods will
cover that. Thanks.
 
Reply With Quote
 
Arne Vajhøj
Guest
Posts: n/a
 
      13th Apr 2012
On 4/12/2012 6:23 PM, Davej wrote:
> On Apr 12, 4:46 pm, Arne Vajhøj<a...@vajhoej.dk> wrote:
>> On 4/12/2012 2:30 PM, Davej wrote:
>>> On Apr 11, 6:51 pm, Davej<galt...@hotmail.com> wrote:
>>>> On Apr 11, 6:30 pm, Arne Vajhøj<a...@vajhoej.dk> wrote:
>>>>> On 4/11/2012 7:22 PM, Davej wrote:
>>>>>> On Apr 11, 6:09 pm, Arne Vajhøj<a...@vajhoej.dk> wrote:
>>>>>>> On 4/11/2012 7:06 PM, Arne Vajhøj wrote:
>>>>>>>> On 4/11/2012 7:01 PM, Davej wrote:

>>
>>>>>>>>> I tried UploadString(), UploadData() and UploadValues() but the page
>>>>>>>>> never seems to see the POST. It just responds with what looks like an
>>>>>>>>> initial rendering.

>>
>>>>>>>> You need to be sure that you are POST'ing to the action URL
>>>>>>>> not the form URL.

>>
>>>>>>> Ooops.

>>
>>>>>>> I just checked your comment.

>>
>>>>>>> //<form name="form1" method="post" action="default.aspx" id="form1">
>>>>>>> //<input name="txtUserID" type="text" id="txtUserID" />
>>>>>>> //<input name="txtPassword" type="password" id="txtPassword" />
>>>>>>> //<input type="submit" name="btnLogin" value="Login" id="btnLogin" />

>>
>>>>>>> If that is an ASP.NET web forms page, then it becomes a bit
>>>>>>> tricky.

>>
>>>>>>> You need to send the viewstate in your POST for things
>>>>>>> to work properly.

>>
>>>>>>> Look at the form source and see the hidden field.

>>
>>>>>> Oh! I forgot about that hidden field! So I need to downloadData() and
>>>>>> then send the name/value pair of the viewstate back with the other
>>>>>> pairs?

>>
>>>>> Yes.

>>
>>>> Bingo. That was it. Much thanks! Should have been obvious to me but I
>>>> just wasn't thinking to look for hidden fields.

>>
>>> I have some working test code now. The routines necessary to extract
>>> the two keys in the hidden fields were rather bothersome.

>>
>> Yes - calling web forms pages programmatically is just cumbersome.
>>
>>> Now I need to figure out how to implement the "non blocking"
>>> methods because this process is pretty slow.

>>
>> Well - you can either use threads/threadpool or go for the asynch
>> methods.

>
> Are there other better schemes that I should be investigating?


I don't think so.

For HTML parsing you could look at HTML Agility Pack.

> I have
> read a little about web services, but they seem to be all about
> exchanging XML data.


Or at least a format intended for an application and not a browser.

> Compared to that the various WebClient methods
> seem pretty powerful. I want to be able to upload and download jpg
> files and some text records. It looks like the WebClient methods will
> cover that.


And if WebClient is not flexible enough then you can fallback to
HttpWebRequest.

Arne

 
Reply With Quote
 
 
 
Reply

Thread Tools
Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Simulating Exchange Server to let 3rd party app pass email to Outl =?Utf-8?B?Q3JhaWc=?= Microsoft Outlook Discussion 0 1st Feb 2005 05:27 PM
CLR error 80004005 & Simulating Mouse Clicks Ong Zihao Windows XP General 1 8th Oct 2004 06:58 PM
How to simulating lightning effects Jan Il Microsoft Powerpoint 2 22nd Mar 2004 06:45 AM
simulating RIS shoba Windows XP Setup 1 16th Dec 2003 03:09 PM
Simulating keyboard input? Mark Stier Windows XP General 0 21st Jul 2003 10:57 PM


Features
 

Advertising
 

Newsgroups
 


All times are GMT +1. The time now is 12:12 AM.