Please help us... POSTing with Http/WebRequest

J

Jonathan Amend

I have noticed that a lot of people were having a similar problem to mine
which involves using the POST method with the
HttpWebRequest/WebRequest/WebClient classes (.NET 1.1). The classes will
send out one HTTP packet that says content of a certain length will follow
but the second packet is never sent, which usually results in a 400 error
from the server. I've tried my code on both HTTP 1.0 and 1.1 servers and
both gave the same result. Here is my example:

Try
Dim LogInURL As String = "http://s2.starkingdoms.com/scripts/main.cgi"
'Dim LogInURL As String =
"http://sklone.homeip.net/scripts/BrowserLogin.aspx"
Dim Account As String = "blah"
Dim Password As String = "blah"

Dim Request As Net.HttpWebRequest = Net.HttpWebRequest.Create(LogInURL)
Request.ContentType = "application/x-www-form-urlencoded"
Request.Method = "POST"
Request.ProtocolVersion = Net.HttpVersion.Version10
'Request.ProtocolVersion = Net.HttpVersion.Version11
Dim RequsetStream As IO.Stream = Request.GetRequestStream()
Dim ASCIIEncoding As New System.Text.ASCIIEncoding
Dim PostData As Byte() = ASCIIEncoding.GetBytes("Account=" & Account &
"Password=" & Password)
RequsetStream.Write(PostData, 0, PostData.Length)
RequsetStream.Close()
Dim Reader As New
IO.StreamReader(Request.GetResponse().GetResponseStream())
Dim ResultHTML As String = Reader.ReadToEnd()
Reader.Close()

MsgBox(ResultHTML)
Catch ex As Exception
MsgBox(ex.Message)
End Try

And here's what happens:

POST /scripts/main.cgi HTTP/1.0
Content-Length: 25
Connection: Keep-Alive
Host: s2.starkingdoms.com


Followed by the Bad Request (400) error.
 
E

EMonaco

Jonathan,

Try commenting out your RequestStream.Close() call... and rerun it.


Erin.
 
E

EMonaco

Jonathan,

Nix that- you have to close the stream. I tried using my HTTP Test tool-
which I am doing my best to complete as well. And I get 400 and this
content:

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">

<HTML><HEAD>

<TITLE>400 Bad Request</TITLE>

</HEAD><BODY>

<H1>Bad Request</H1>

Your browser sent a request that this server could not understand.<P>

The request line contained invalid characters following the protocol
string.<P>

<P>

<HR>

<ADDRESS>Apache/1.3.27 Server at starkingdoms.com Port 80</ADDRESS>

</BODY></HTML>



Basically what its saying (I think) is that it does not understand the s2.
before starkingdoms.com, removing that, it says there is no
/scripts/main.cgi at starkingdoms.com....

Erin.;
 
F

furty

In the code sample supplied, you are not setting the ContentLength property
for the HttpWebRequest, this is more than likely the problem. Try changing
your code as follows:

Dim PostData As Byte() = ASCIIEncoding.GetBytes("Account=" & Account &
"Password=" & Password)
Request.ContentLength = PostData.Length
RequestStream.Write(PostData, 0, PostData.Length)

Now if you happen to find out why you cannot POST data to HTTP 1.0 servers,
I'd be very appreciative!

Furty.

PS. My VB syntax could be wrong, I haven't coded in Basic since the
Commodore 64 :)
 
E

EMonaco

That still returns a 400 error. Is it possible the form data is just not
right and the cgi script is returning the error?


Erin
 
J

Jonathan Amend

OK I missed the & before the second variable pair but that still doesn't fix
the problem. Not closing the stream would obviously not send the request so
that doesn't fix it either. ContentLength sets itself automatically once you
put the PostData into the stream. Remember, I'm not the only one having this
problem so even if there is a mistake in my coding somewhere I doubt all the
other people made the same one. It could be that it's OS-related (I'm
running Windows Server 2003) but I couldn't find anything to suggest that.
 
E

EMonaco

Jonathan,

I am on XP Pro. If I use "blah" for username & password I get the 400
error too. Have you confirmed that your username, password actually work (by
going to the site through IE, netscape, or whatever)?
 
J

Jonathan Amend

Yes... I still get a 400 error if I try to log in with a real account and
password. The code does work on my own server
(http://sklone.homeip.net/scripts/BrowserLogin.aspx) so it probably has
something to do with HTTP 1.0 which was mentioned in similar threads. The
project I'm working on is a specialized browser for starkingdoms which
handles the popups. I originally wrote it in VB6 using the INET control so
now I'm trying to port it to VB.NET 1.1 and the System.Net namespace. I can
get it to work if I use a wrapped INET control but if I use that I might as
well stick with VB6 altogether. I'll set up an XP box later and see what
happens if I run it on there.
 
F

furty

If you're not doing it already, use a HTTP sniffer or Proxy to monitor the
headers and body data between the server and your system. Login using your
regular browser first, then run your application and compare the differences
between the data. You may find something like my current issue, where
HttpWebRequest always sends an "Expect: 100-continue" header with POST
requests, even for HTTP 1.0 servers that will choke on it.

If you don't have a sniffer or proxy, I can recommend Stephen Toub's
SSLProxy, available from GotDotNet in the user samples area. It's written in
C# and easily extensible for almost any usage - I have modified it to save
all body data in POSTs, and it has proved invaluable for debugging my
projects.

I ran that website through the SSL Proxy with a dummy username and password,
with the following output:

REQUEST: POST http://s2.starkingdoms.com/scripts/main.cgi HTTP/1.0
RESPONSE: HTTP/1.1 302 Found
DATE: 12/08/2003 4:16:10 AM GMT

HEADERS FROM CLIENT TO PROXY:
-----------------------------
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg,
application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword,
application/x-shockwave-flash, */*
Referer: http://s2.starkingdoms.com/login.php
Accept-Language: en-au
Content-Type: application/x-www-form-urlencoded
Proxy-Connection: Keep-Alive
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR
1.0.3705; .NET CLR 1.1.4322)
Host: s2.starkingdoms.com
Content-Length: 37
Pragma: no-cache


BODY FROM CLIENT TO PROXY:
-----------------------------
Account=testAccount&Password=testPass

HEADERS FROM SERVER TO PROXY:
-----------------------------
Date: Tue, 12 Aug 2003 04:16:32 GMT
Server: Apache/1.3.27 (Unix) (Red-Hat/Linux) mod_ssl/2.8.12 OpenSSL/0.9.6b
DAV/1.0.2 PHP/4.2.3 mod_perl/1.26 mod_throttle/3.1.2
P3P: CP="NOI DSP COR ADMa TAIa UNRa NOR"
Location: http://s2.starkingdoms.com/login.php?error=Account not found. What
this means:<br>1. The account you entered in was wrong.<br>2. Your account
has been killed. If this is true you should have received an email stating
that your account died.<br>3. The account was erased on the last reset on
July 3, 2002.<br>4. There was a server error that removed the account list.
In such case it will be repaired soon.
Connection: close
Content-Type: text/html; charset=iso-8859-1
 
E

EMonaco

Furty,

I got Jonathan's example to work on XP Pro- as such it was sending Expect:
100-continue as well.
I do agree. I'd definitely go with an HTTPProxy that captures the request
and response info.

Thats a lot of headers, what client were you using?


Erin.
 
F

furty

That's quite atypical of header info in my experience over the years - for
the record I'm using a very standard setup: WinXP SP1, IE 6 SP1, .Net 1.0,
..Net 1.1 & Office XP - all of which is readily disocovered from my client
headers :)

HTTP code 400 means bad request - it's just a guess, but I fugure that's
what Apache throws when it sees headers it doesn't like, whereas IIS throws
a 500 server error code. The good news is that the site in question accepts
HTTP 1.1, which means that the Expect: 100-continue header issue shouldn't
cause any problems.
 
J

Jonathan Amend

I already use a program that monitors winsock activity for a certain
process. I tried it on Windows XP and it still doesn't work. You used VS.NET
2003 and .NET 1.1 right? I think it has something to do with .NET's support
for HTTP 1.0 or Apache's ability to handle the Expect: part of the header.

HTTP 1.0 with starkingdoms (Apache):

POST /scripts/main.cgi HTTP/1.1
Content-Type: application/x-www-form-urlencoded
Content-Length: 32
Expect: 100-continue
Connection: Keep-Alive
Host: s1.starkingdoms.com

GET /login.php?error=Account not found. What this means:<br>1. The account
you entered in was wrong.<br>2. Your account has been killed. If this is
true you should have received an email stating that your account died.<br>3.
The account was erased on the last reset.<br>4. There was a server error
that removed the account list. In such case it will be repaired soon.
HTTP/1.0
Content-Type: application/x-www-form-urlencoded
Connection: Keep-Alive
Host: 66.250.109.245


HTTP 1.1 with sklone (my ASP.NET starkingdoms-clone on Windows Server 2003)

POST /scripts/BrowserLogin.aspx HTTP/1.1
Content-Type: application/x-www-form-urlencoded
Content-Length: 26
Expect: 100-continue
Host: sklone.homeip.net

It logs in fine (BrowserLogin.aspx returns "1")
 
F

feroze

Could it be that there is a proxy in between you and the server, which is
mangling the messages ?

Can you send the same request using a socket based client (i.e outside of
webrequest) and get it to succeed ?

feroze
 
J

Jonathan Amend

There is no proxy... Like I said before my project is a port from an old VB6
app which used INET and it works. I can also get the .NET version to work
using a wrapped INET control but that's useless.
 
F

feroze

Since it looks like the cgi script is using a FORMS POST, can you try
something ? Use System.Net.WebClient() and use WebClient.UploadValues() to
upload your data. With three lines of code, you should be able to see if it
will work or not. I personally dont see anything wrong with the code you
posted. It could be that the CGI script on the server is choking.
Apache/1.3.2* were all http/1.1 compliant servers.

If you were using a client protocol of 1.0 to do the upload, then you might
be running into the bug whereby the lib was sendign a "Expect: 100-Continue"
header. You can download a QFE from microsoft.com to fix that issue.

=========================
THis posting is provided as-is. It provides no warranties and confers no
rights.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Similar Threads

Using WebClient/WebRequest/WebResponse to post postdata and get the result 4
Accessing encrypted Site with Net.HttpWebRequest 1
Posting login data with HttpWebRequest 2
Help with WebRequest Authentication 1
Invalid Characters, Posting XML Data, Webrequest 1
Help with HttpWebRequest.Method 2
Net.WebRequest/WebResponse ?'s 1
Posting multipart/form-data using VB.NET 7

Top