HttpWebRequest/.ServerXMLHTTP.4.0 - different results

  • Thread starter Thread starter Guest
  • Start date Start date
G

Guest

I am attempting to post to a url (https://FakeURL/logon.asp) using the
HttpWebRequest class. The response for a succesful post will contain
the html for the logon user's default page.

We've accomplished this in the past utilizing the ServerXMLHTTP object. When
I try an equivalent? post utilizing the HttpWebRequest class,
the response contains the html for logon.asp (the same page that was posted
to), which indicates to me that in some way my post didn't work.
No exceptions are thrown, it's just that the wrong page is returned.



The request is through a proxy server to the internet, via https. I've set
the headers the same way as with the successful call through
ServerXMLHTTP. I've tried it both with setting and not setting the
Credentials property of the request. In setting Credentials, I've tried
using a CredentialCache, or just a NetworkCredential

The code for both the ServerXMLHTTP, and the .Net page is below. Can anyone
tell me what I'm missing?

Thanks,


Dave Brown



'Dot Net Code
-------------------------------------------------------------------------------------------

Public Class WebForm1

Inherits System.Web.UI.Page

#Region " Web Form Designer Generated Code "

'This call is required by the Web Form Designer.
<System.Diagnostics.DebuggerStepThrough()> Private Sub
InitializeComponent()

End Sub

'NOTE: The following placeholder declaration is required by the Web Form
Designer.
'Do not delete or move it.
Private designerPlaceholderDeclaration As System.Object

Private Sub Page_Init(ByVal sender As System.Object, ByVal e As
System.EventArgs) Handles MyBase.Init
'CODEGEN: This method call is required by the Web Form Designer
'Do not modify it using the code editor.
InitializeComponent()
End Sub

#End Region

Private Sub Page_Load(ByVal sender As System.Object, ByVal e As
System.EventArgs) Handles MyBase.Load
Dim sUID As String = "xxxxx"
Dim sPassword As String = "xxxxx"
Dim strURL As String = "https://FakeURL/logon.asp"
Dim strRetval As String
strRetval = getPage(strURL, "Mode=&Username=" & sUID & "&Password="
& sPassword)
Response.Write(strRetval)
End Sub
Public Shared Function getPage(ByVal url As String, ByVal payload As
String) As String
Dim result As WebResponse
Dim proxyObject As New WebProxy("proxy.fake.fake.com:8080", False)


Try

Dim RequestStream As Stream
Dim ReceiveStream As Stream
Dim encode As Encoding
Dim sr As StreamReader

'Convert the web request to an HttpWebRequest so the UserAgent
can be set
'Don't know if it makes a difference, but the xmlServerHttp
object did this
'so might as well try it here
Dim req As HttpWebRequest = CType(WebRequest.Create(url),
HttpWebRequest)

req.Method = "POST"
req.ContentType = "application/x-www-form-urlencoded"
req.UserAgent = "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT
5.0)"
req.Proxy = proxyObject

'Tried this but it didn't seem to matter either way
'ensure that it sends default credentials
'req.PreAuthenticate = True

'Dim MyCredential As New System.Net.NetworkCredential("xxxxx",
"xxxxx")

'req.Credentials = MyCredential
'System.Net.CredentialCache.DefaultCredentials

'Dim cache As CredentialCache = New CredentialCache
'cache.Add(New Uri("https://FakeURL/logon.asp"), "Basic", New
NetworkCredential("xxxxx", "xxxxx"))
'req.Credentials = cache



' ----- This block is straight from the .Net samples
----------------------------------------------
Dim SomeBytes() As Byte
Dim UrlEncoded As New StringBuilder
Dim reserved() As Char = {ChrW(63), ChrW(61), ChrW(38)}

If payload <> Nothing Then
Dim i As Integer = 0
Dim j As Integer
While i < payload.Length
j = payload.IndexOfAny(reserved, i)
If j = -1 Then

UrlEncoded.Append(HttpUtility.UrlEncode(payload.Substring(i, payload.Length -
i)))
Exit While
End If

UrlEncoded.Append(HttpUtility.UrlEncode(payload.Substring(i, j - i)))
UrlEncoded.Append(payload.Substring(j, 1))
i = j + 1
End While
SomeBytes =
System.Text.Encoding.UTF8.GetBytes(UrlEncoded.ToString())
req.ContentLength = SomeBytes.Length
RequestStream = req.GetRequestStream()
RequestStream.Write(SomeBytes, 0, SomeBytes.Length)
RequestStream.Close()
Else
req.ContentLength = 0
End If

'---------------------------------------------------------------------------------------------------

'Tried this as an alternative forumulation to the block directly
above
'Dim requestBytes() As Byte
'requestBytes = System.Text.Encoding.ASCII.GetBytes(payload)
'RequestStream = req.GetRequestStream()
'RequestStream.Write(requestBytes, 0, requestBytes.Length)

'---------------------------------------------------------------------------------------------------


result = req.GetResponse()
ReceiveStream = result.GetResponseStream()
encode = System.Text.Encoding.GetEncoding("utf-8")
sr = New StreamReader(ReceiveStream, encode)



Dim read(256) As Char
Dim count As Integer = sr.Read(read, 0, 256)

Dim strbRetval As New System.Text.StringBuilder
Do While count > 0
Dim str As String = New String(read, 0, count)
strbRetval.Append(str)
count = sr.Read(read, 0, 256)
Loop
Return strbRetval.ToString()
Catch Exc As Exception
Return Exc.Message
Finally
If Not result Is Nothing Then
result.Close()
End If
End Try
End Function


End Class
'-------------------------------------------------------------------------------------------------------------------

'ServerXMLHTTP Code
------------------------------------------------------------------------------------

Dim lResolve As Long
Dim lConnect As Long
Dim lSend As Long
Dim lReceive As Long
Dim xmlServerHTTP


xmlServerHTTP = CreateObject("MSXML2.ServerXMLHTTP.4.0")


lResolve = 60000
lConnect = 60000
lSend = 60000
lReceive = 120000

xmlServerHttp.setProxy(2, "proxy.fake.fake.com:8080", "")
xmlServerHttp.open("POST", "https://FakeURL/logon.asp", False)
xmlServerHttp.setRequestHeader("User-Agent", "Mozilla/4.0 (compatible; MSIE
5.5; Windows NT 5.0)")
xmlServerHttp.setRequestHeader("Content-Type",
"application/x-www-form-urlencoded")
xmlServerHttp.send("Mode=&Username=" & sUID & "&Password=" & sPassword)

While xmlServerHTTP.readyState <> 4
xmlServerHTTP.waitForResponse 1000
Wend
'----------------------------------------------------------------------------------------------------------
 
Ok, I think this is the issue

The ServerXMLHTTP.4.0 makes an http 1.1 protocol request, and does not
include/set the "Expect" header. The HttpWebRequest class makes an http
1.1 request by default, and does not allow you to delete or modify the Expect
header. Even if you set MyReq.Expect = "", by the time it gets to the Server
it changes to 100-Continue. The only way to get rid of the Expects header is
to change MyReq.ProtocolVersion = HttpVersion.Version10.

It looks like this behavior is by design to be in accordance with rfc2616
http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html.

However, if this is the case, in making this design choice, Microsoft has
sacrificed the "real world" usability of the class. Even the rfc states that
"Many older HTTP/1.0 and HTTP/1.1 applications do not understand the Expect
header."

If I have this wrong, or have missed something I look forward to someone
correcting me.
 
Dave said:
Ok, I think this is the issue

The ServerXMLHTTP.4.0 makes an http 1.1 protocol request, and does
not include/set the "Expect" header. The HttpWebRequest class makes
an http 1.1 request by default, and does not allow you to delete or
modify the Expect header. Even if you set MyReq.Expect = "", by the
time it gets to the Server it changes to 100-Continue. The only way
to get rid of the Expects header is to change MyReq.ProtocolVersion =
HttpVersion.Version10.

It looks like this behavior is by design to be in accordance with
rfc2616 http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html.

However, if this is the case, in making this design choice, Microsoft
has sacrificed the "real world" usability of the class. Even the rfc
states that "Many older HTTP/1.0 and HTTP/1.1 applications do not
understand the Expect header."

You can turn off Expectations for a specific connection using
ServicePoint.Expect100Continue or globally using
ServicePointManager.Expect100Continue.
If I have this wrong, or have missed something I look forward to
someone correcting me.

See also my other reply. I'm sure your problem is not about
Expectations.

Cheers,
 
Joerg,

Thanks for your replies. Your point about the cookies is well taken,
however the response I'm getting from the server (redisplay of the logon page
rather then a
redirect to a default page) indicates that my session cookie has not been
set yet.

I'll look into ServicePoint and ServicePointManager, and post back whether
it works.
In the meantime, I've had to drop back to .asp to get the job done...

Thanks again,
Dave
 
Dave said:
Joerg,

Thanks for your replies. Your point about the cookies is well taken,
however the response I'm getting from the server (redisplay of the
logon page rather then a
redirect to a default page) indicates that my session cookie has not
been set yet.

Why? Redirecting to the logon page is a web app's normal behaviour when
you're not logged on.

Cheers,
 
Joerg,

I agree with you that redirection to the logon page in the event of a failed
logon is
normal behavior. However, the redirection in this case is from POST where I
attempted to post a valid user name and password. Had this POST succeeded,
then
I think the apps normal behavior would be to set some sort of session
cookie, at which point my code would have to worry about it, as you pointed
out. Since the
POST didn't succeed, I'm assuming there is no cookie to consume. Although
the app
may have set a cookie indicating a login failure, I thought the larger
problem
was the login failure itself, and that the cookie problem was only worth
dealing with if the login problem was solved.

Thanks,
 
Joerge,

You were right about the cookies all along. I was logging in succesfully,
but since the cookie was not passed to the redirect page, it threw me back to
the login page.
Since MSXML was caching the cookies for me, it was never an issue there.

I'm wondering how many other "gotchas" are lurking in the .net implementation
of this functionality. It took me 3 days in .net to get to the same point I
got to
with MSXML in 10 minutes. The .net class has about 3 times the amount of
code as the .asp function. Is this truly progress?
 
Dave said:
Joerg,

I agree with you that redirection to the logon page in the event of a
failed logon is
normal behavior. However, the redirection in this case is from POST
where I attempted to post a valid user name and password. Had this
POST succeeded, then
I think the apps normal behavior would be to set some sort of session
cookie, at which point my code would have to worry about it, as you
pointed out.

I see. I missed the fact that you're running into the problem at logon,
not after logon.

Since the
POST didn't succeed, I'm assuming there is no cookie to consume.
Although the app
may have set a cookie indicating a login failure, I thought the
larger problem
was the login failure itself, and that the cookie problem was only
worth dealing with if the login problem was solved.

That's true.

Cheers,
 
Dave said:
Joerge,

You were right about the cookies all along. I was logging in
succesfully, but since the cookie was not passed to the redirect
page, it threw me back to the login page.
Since MSXML was caching the cookies for me, it was never an issue
there.

Morale of the story: Let there be a CookieContainer. I really thought
about a redirect after logon scenario yesterday after reading your last
response, but wasn't in the mood for testing ;-)
I'm wondering how many other "gotchas" are lurking in the .net
implementation of this functionality. It took me 3 days in .net to
get to the same point I got to
with MSXML in 10 minutes. The .net class has about 3 times the amount
of code as the .asp function. Is this truly progress?

Remember that there's also System.Net.WebClient, which works fine for
many simple HTTP operations, although it still doesn't support
CookieContainers and is thus useless for programmatically accessing web
sites.


Cheers,
 
Back
Top