Automating a POST request

[James Johnson]

Dear C#Dex,

I am trying to automate a POST to a web page that clicks a button. I
have been able to hit a target web page and run the web page. However,
the button on the page does not click. I can set the target web page to
change to a new URL when I hit it, and that works, but I cannot get the
button to click based on my POST command. Is there some secret to
clicking the button? I have a button named btnGo and I set the value of
btnGo=clicked; however, the button is not clicked. I can, for example,
provide querystring values and the target web page detects these and
will perform logic (such as navigating to another page) based upon the
provided query string, but I can’t get the button to click using a POST
command. The relevant code is shown below:

url= "http://localhost/FirstProject/WebForm2.aspx";
payload = "txtUserID=text&txtPassword=text&btnGo=clicked&";

WebRequest req = WebRequest.Create(url);
req.Method = "POST";
req.ContentType = "application/x-www-form-urlencoded";

StringBuilder UrlEncoded = new StringBuilder();
Char[] reserved = {'?', '=', '&'};
byte[] SomeBytes = null;

if (payload != null)
{
int i=0, j;
while(i<payload.Length)
{
j=payload.IndexOfAny(reserved, i);
if (j==-1)
{
UrlEncoded.Append(HttpUtility.UrlEncode(payload.Substring(i,
payload.Length-i))); //URL encodes and appends to the contents of the
stringbuilder any text remaining in the payload after the last special
char
break;
}
UrlEncoded.Append(HttpUtility.UrlEncode(payload.Substring(i,
j-i))); //appends URL encoded text that is between reserved characters
to the stringbuilder contents
UrlEncoded.Append(payload.Substring(j,1)); //appends literal
reserved char to contents of stringbuilder
i = j+1; //increments index to get text starting point at
position past the position of the current reserved character
}

string str = UrlEncoded.ToString();
SomeBytes = Encoding.UTF8.GetBytes(UrlEncoded.ToString()); //Encodes
URL string into hexadecimal bytes
req.ContentLength = SomeBytes.Length; //Gets length of string
encoded into bytes that is to be sent as a POST request

//The URL plus the encoded variables
Stream newStream = req.GetRequestStream(); //Create a Stream variable
so we can write it to the Http stream and initialize it with the encoded
stream
newStream.Write(SomeBytes, 0, SomeBytes.Length); //Writes Http stream
to IP address returned by the URI
newStream.Close(); //Free up resources by closing connection
}
else
{
req.ContentLength = 0;
}

result = req.GetResponse(); //WebResponse get response of what
happened as a result of the POST that you just wrote to the URI—-The
button is not clicked--the page stays the same instead of navigating to
the new page!

Stream ReceiveStream = result.GetResponseStream();
Encoding encode = System.Text.Encoding.GetEncoding("utf-8");
StreamReader sr = new StreamReader( ReceiveStream, encode );
Console.WriteLine("\r\nResponse stream received");
Char[] read = new Char[256];
int count = sr.Read( read, 0, 256 );
Console.WriteLine("HTML...\r\n");
while (count > 0)
{
String str = new String(read, 0, count);
Console.Write(str);
count = sr.Read(read, 0, 256);
}
Console.WriteLine("");

Do you have any suggestions as to why the button won't click? Thanks
for your help.

James J.

 

[Nick Malik]

You ask an odd question, "why won't the button click." But that isn't your
problem.

You are composing a complicated POST request. You've taken great pains to
create a string of parameters, then go through and encode the values in the
parameters without encoding the seperators. You then submit this request to
a server. The results aren't what you expect.

It is fairly clear to me that the server doesn't "like" the POST. It is not
accepting the data you are sending, so it is returning the form page rather
than returning the next page you expect.

Two suggestions:
1) The way you are creating your payload string is nuts.
payload = "txtUserID=" + HttpUtility.UrlEncode(text) + "&txtPassword="
+ HttpUtility.UrlEncode(text) + "&btnGo=clicked&";
this will do the same thing. Note: I do not remember, off the top of
my head, if this is going to be considered a valid string by the web server.

2) Look at the cookies. It looks to me like you are trying to get past a
login page. Most sites handle login by sending a cookie on the login form.
The cookie needs to come back on the Post request. If the login is
successful, other cookies will come back on the response page. You have to
append ALL of these cookies to every subsequent request to the web server.

This is how ASP and ASP.NET work as well. If you intend to login to a web
site that uses cookies for session management, and a large number of sites
do this, you will not be successful until you handle the cookies.

Good Luck,

--- Nick

Dear C#Dex,

I am trying to automate a POST to a web page that clicks a button. I
have been able to hit a target web page and run the web page. However,
the button on the page does not click. I can set the target web page to
change to a new URL when I hit it, and that works, but I cannot get the
button to click based on my POST command. Is there some secret to
clicking the button? I have a button named btnGo and I set the value of
btnGo=clicked; however, the button is not clicked. I can, for example,
provide querystring values and the target web page detects these and
will perform logic (such as navigating to another page) based upon the
provided query string, but I can't get the button to click using a POST
command. The relevant code is shown below:

url= "http://localhost/FirstProject/WebForm2.aspx";
payload = "txtUserID=text&txtPassword=text&btnGo=clicked&";

WebRequest req = WebRequest.Create(url);
req.Method = "POST";
req.ContentType = "application/x-www-form-urlencoded";

StringBuilder UrlEncoded = new StringBuilder();
Char[] reserved = {'?', '=', '&'};
byte[] SomeBytes = null;

if (payload != null)
{
int i=0, j;
while(i<payload.Length)
{
j=payload.IndexOfAny(reserved, i);
if (j==-1)
{
UrlEncoded.Append(HttpUtility.UrlEncode(payload.Substring(i,
payload.Length-i))); //URL encodes and appends to the contents of the
stringbuilder any text remaining in the payload after the last special
char
break;
}
UrlEncoded.Append(HttpUtility.UrlEncode(payload.Substring(i,
j-i))); //appends URL encoded text that is between reserved characters
to the stringbuilder contents
UrlEncoded.Append(payload.Substring(j,1)); //appends literal
reserved char to contents of stringbuilder
i = j+1; //increments index to get text starting point at
position past the position of the current reserved character
}

string str = UrlEncoded.ToString();
SomeBytes = Encoding.UTF8.GetBytes(UrlEncoded.ToString()); //Encodes
URL string into hexadecimal bytes
req.ContentLength = SomeBytes.Length; //Gets length of string
encoded into bytes that is to be sent as a POST request

//The URL plus the encoded variables
Stream newStream = req.GetRequestStream(); //Create a Stream variable
so we can write it to the Http stream and initialize it with the encoded
stream
newStream.Write(SomeBytes, 0, SomeBytes.Length); //Writes Http stream
to IP address returned by the URI
newStream.Close(); //Free up resources by closing connection
}
else
{
req.ContentLength = 0;
}

result = req.GetResponse(); //WebResponse get response of what
happened as a result of the POST that you just wrote to the URI--The
button is not clicked--the page stays the same instead of navigating to
the new page!

Stream ReceiveStream = result.GetResponseStream();
Encoding encode = System.Text.Encoding.GetEncoding("utf-8");
StreamReader sr = new StreamReader( ReceiveStream, encode );
Console.WriteLine("\r\nResponse stream received");
Char[] read = new Char[256];
int count = sr.Read( read, 0, 256 );
Console.WriteLine("HTML...\r\n");
while (count > 0)
{
String str = new String(read, 0, count);
Console.Write(str);
count = sr.Read(read, 0, 256);
}
Console.WriteLine("");

Do you have any suggestions as to why the button won't click? Thanks
for your help.

James J.

 

[Scott Allen]

Hi James:

I have an article that might help you out:

Screen Scraping, ViewState, and Authentication using ASP.Net
http://odetocode.com/Articles/162.aspx


You need to construct your POST string carefully, and perhaps compare
what you are sending to the server versus what a browser sends to the
server using a tool like Fiddler which I link to in the article. This
is one good way of finding bugs in these types of applications.

If you are trying to login to a site, you'll probably also have to
deal with cookies, which the article also demonstrates.

HTH,

--
Scott
http://www.OdeToCode.com

Dear C#Dex,

I am trying to automate a POST to a web page that clicks a button. I
have been able to hit a target web page and run the web page. However,
the button on the page does not click. I can set the target web page to
change to a new URL when I hit it, and that works, but I cannot get the
button to click based on my POST command. Is there some secret to
clicking the button? I have a button named btnGo and I set the value of
btnGo=clicked; however, the button is not clicked. I can, for example,
provide querystring values and the target web page detects these and
will perform logic (such as navigating to another page) based upon the
provided query string, but I can’t get the button to click using a POST
command. The relevant code is shown below:

url= "http://localhost/FirstProject/WebForm2.aspx";
payload = "txtUserID=text&txtPassword=text&btnGo=clicked&";

WebRequest req = WebRequest.Create(url);
req.Method = "POST";
req.ContentType = "application/x-www-form-urlencoded";

StringBuilder UrlEncoded = new StringBuilder();
Char[] reserved = {'?', '=', '&'};
byte[] SomeBytes = null;

if (payload != null)
{
int i=0, j;
while(i<payload.Length)
{
j=payload.IndexOfAny(reserved, i);
if (j==-1)
{
UrlEncoded.Append(HttpUtility.UrlEncode(payload.Substring(i,
payload.Length-i))); //URL encodes and appends to the contents of the
stringbuilder any text remaining in the payload after the last special
char
break;
}
UrlEncoded.Append(HttpUtility.UrlEncode(payload.Substring(i,
j-i))); //appends URL encoded text that is between reserved characters
to the stringbuilder contents
UrlEncoded.Append(payload.Substring(j,1)); //appends literal
reserved char to contents of stringbuilder
i = j+1; //increments index to get text starting point at
position past the position of the current reserved character
}

string str = UrlEncoded.ToString();
SomeBytes = Encoding.UTF8.GetBytes(UrlEncoded.ToString()); //Encodes
URL string into hexadecimal bytes
req.ContentLength = SomeBytes.Length; //Gets length of string
encoded into bytes that is to be sent as a POST request

//The URL plus the encoded variables
Stream newStream = req.GetRequestStream(); //Create a Stream variable
so we can write it to the Http stream and initialize it with the encoded
stream
newStream.Write(SomeBytes, 0, SomeBytes.Length); //Writes Http stream
to IP address returned by the URI
newStream.Close(); //Free up resources by closing connection
}
else
{
req.ContentLength = 0;
}

result = req.GetResponse(); //WebResponse get response of what
happened as a result of the POST that you just wrote to the URI—-The
button is not clicked--the page stays the same instead of navigating to
the new page!

Stream ReceiveStream = result.GetResponseStream();
Encoding encode = System.Text.Encoding.GetEncoding("utf-8");
StreamReader sr = new StreamReader( ReceiveStream, encode );
Console.WriteLine("\r\nResponse stream received");
Char[] read = new Char[256];
int count = sr.Read( read, 0, 256 );
Console.WriteLine("HTML...\r\n");
while (count > 0)
{
String str = new String(read, 0, count);
Console.Write(str);
count = sr.Read(read, 0, 256);
}
Console.WriteLine("");

Do you have any suggestions as to why the button won't click? Thanks
for your help.

James J.

 

[Joerg Jooss]

Nick Malik wrote:
[...]

Two suggestions:
1) The way you are creating your payload string is nuts.
payload = "txtUserID=" + HttpUtility.UrlEncode(text) +
"&txtPassword=" + HttpUtility.UrlEncode(text) + "&btnGo=clicked&";
this will do the same thing. Note: I do not remember, off the
top of my head, if this is going to be considered a valid string by
the web server.

Careful. HttpUtility.UrlEncode() does not work on complete query strings, it
just processes words:

string query1 = "txtUserID=" + HttpUtility.UrlEncode("foo");
Console.WriteLine(query1);

% txtUserID=foo

string query2 = HttpUtility.UrlEncode("txtUserID=foo");
Console.WriteLine(query2);

% txtUserID%3dfoo

Cheers,

 

[James Johnson]

Dear Nick,

In reply to

1) The way you are creating your payload string is nuts.

I copied it from MS sample code. Maybe they were trying to avoid the
HttpUtility.UrlEncode() does not work on complete query strings problem
that Joerg noticed.

I am also wondering what modifications I would have to make in my code
to make the approach work with a plain HTML page. Would I need to
change the POST to GET? I am sure I would not need the viewstate
parameter. I don't think a plain HTML page would process the .Net
methods that the sample code uses.

Thanks,

James J.