I have been working through an issue with a client for some time, and would
like to see if anyone can confirm my observations of behavior experienced
when using the HttpWebRequest class.

Here is the scenario:
The client application is written in C# and is making a large HTTP POST
request to a backend web server, using SSL. The payload contains MIME
attachments (this is a Soap with Attachments client), of which the last
attachment is a large, binary, blob (the original file that the blob is
constructed from can be pretty big – 8, 10 even 25 MB).

We are able to send these large requests, when SSL is turned off, but as
soon as SSL is turned back on, the largest POST we can make contains an
attachment of about 4MB. Any payloads that contain attachments greater than
4MB end up timing out on the server.

My observations:
From what I have read, it appears that the HttpWebRequest class sends a POST
by breaking up the headers and the body and sending them seperately (in the
same connection). Thus, the headers are sent to the server and then the body
follows in a different transmission. My backend server seems to reflect this
as well – in the cases where the server times out, the headers are processed
correctly, as noted by the server logs. But the time out occurs as the server
waits for the body to be sent to it.

I have verified with the server vendor that the server software does indeed
have a 10 second timeout, where if no data has come across in that 10 second
interval, the server aborts the handling of the request and shuts down the
connection. On the client, I have done some rudimentary sniffing and it
appears that indeed there is no traffic happening during this 10 second
interval (after sending the headers). I am making an assumption that with a
large request body like this, under SSL, the client/framework needs to do
some work to prepare the body (like applying the appropriate encryption) and
this is taking a period of time greater than 10 seconds to complete. I have
somewhat proven this by sending the request to a proxy first, and the proxy
takes a good 30secs – 2 mins to fully receive the request from the client,
before being ready to transmit it to the server. Once the proxy receives the
full request, I am able to forward it to the backend server successfully –
which tells me that the format of the payload is correct.


What I am trying to figure out is the following:
- Is the HttpWebRequest misbehaving?
- Is something coded incorrectly in the client that is causing this delay in
transmission?
- Is the backend server in the wrong for timing out an active request in
such a short period of time?

If this is simply the normal behavior of HttpWebRequest, I am inclined to
recommend that the developer look at using the SSLStream and TcpClient
classes in the 2.0 Framework, which would give him complete control over
everything happening at the socket level. But I’d like to confirm that there
is nothing that we can change in the current HttpWebRequest client, before
heading down that path.

Thanks in advance for your response – I’d love to determine some sort of
resolution on this topic.

Jeff Killberg wrote:

> I have been working through an issue with a client for some time, and
> would like to see if anyone can confirm my observations of behavior
> experienced when using the HttpWebRequest class.
>
> Here is the scenario:
> The client application is written in C# and is making a large HTTP
> POST request to a backend web server, using SSL. The payload contains
> MIME attachments (this is a Soap with Attachments client), of which
> the last attachment is a large, binary, blob (the original file that
> the blob is constructed from can be pretty big – 8, 10 even 25 MB).
>
> We are able to send these large requests, when SSL is turned off, but
> as soon as SSL is turned back on, the largest POST we can make
> contains an attachment of about 4MB. Any payloads that contain
> attachments greater than 4MB end up timing out on the server.
[...]
> What I am trying to figure out is the following:
> - Is the HttpWebRequest misbehaving?

No. Using HTTP Expectations is actually a Good Thing assuming you need
to reject certain POST data due to size limitations.

> - Is something coded incorrectly in the client that is causing this
> delay in transmission?

Well, if you don't want Expectations at all, you can turn them off:

// Do that before creating HttpWebRequest objects
ServicePointManager.Expect100Continue = false;

> - Is the backend server in the wrong for timing out an active request
> in such a short period of time?

10 secs seems a tad aggressive to me, but I'd expect that to be
configurable for your web server -- it isn't?

Cheers,
--
http://www.joergjooss.de
mailto:news-(E-Mail Removed)

Thanks for the reply, Joerg.

> Well, if you don't want Expectations at all, you can turn them off:
>
> // Do that before creating HttpWebRequest objects
> ServicePointManager.Expect100Continue = false;
I'll give a little bit more context - I am currently turning off
Expectations (using the same code as you show below) as well as forcing the
1.0 protocol in my request.


> 10 secs seems a tad aggressive to me, but I'd expect that to be
> configurable for your web server -- it isn't?
I completely agree - but the server is housed in a hardware appliance, and
the vendor says that this value is non-configurable.



"Joerg Jooss" wrote:

> Jeff Killberg wrote:
>
> > I have been working through an issue with a client for some time, and
> > would like to see if anyone can confirm my observations of behavior
> > experienced when using the HttpWebRequest class.
> >
> > Here is the scenario:
> > The client application is written in C# and is making a large HTTP
> > POST request to a backend web server, using SSL. The payload contains
> > MIME attachments (this is a Soap with Attachments client), of which
> > the last attachment is a large, binary, blob (the original file that
> > the blob is constructed from can be pretty big – 8, 10 even 25 MB).
> >
> > We are able to send these large requests, when SSL is turned off, but
> > as soon as SSL is turned back on, the largest POST we can make
> > contains an attachment of about 4MB. Any payloads that contain
> > attachments greater than 4MB end up timing out on the server.
> [...]
> > What I am trying to figure out is the following:
> > - Is the HttpWebRequest misbehaving?
>
> No. Using HTTP Expectations is actually a Good Thing assuming you need
> to reject certain POST data due to size limitations.
>
> > - Is something coded incorrectly in the client that is causing this
> > delay in transmission?
>
> Well, if you don't want Expectations at all, you can turn them off:
>
> // Do that before creating HttpWebRequest objects
> ServicePointManager.Expect100Continue = false;
>
> > - Is the backend server in the wrong for timing out an active request
> > in such a short period of time?
>
> 10 secs seems a tad aggressive to me, but I'd expect that to be
> configurable for your web server -- it isn't?
>
> Cheers,
> --
> http://www.joergjooss.de
> mailto:news-(E-Mail Removed)
>

Jeff Killberg wrote:

> Thanks for the reply, Joerg.
>
> > Well, if you don't want Expectations at all, you can turn them off:
> >
> > // Do that before creating HttpWebRequest objects
> > ServicePointManager.Expect100Continue = false;
> I'll give a little bit more context - I am currently turning off
> Expectations (using the same code as you show below) as well as
> forcing the 1.0 protocol in my request.

You shouldn't need to do both -- Expectations only exist in HTTP 1.1.
Actually, I would never downgrade to HTTP 1.0, due to the severe
limitations regarding cache control, redirects, yada yada...

>
> > 10 secs seems a tad aggressive to me, but I'd expect that to be
> > configurable for your web server -- it isn't?
> I completely agree - but the server is housed in a hardware
> appliance, and the vendor says that this value is non-configurable.

But either of the solutions discussed above does the trick, don't they?

Cheers,
--
http://www.joergjooss.de
mailto:news-(E-Mail Removed)

> But either of the solutions discussed above does the trick, don't they?
Unfortunately, no. I spent a large amount of time trying different
combinations of 1.0 and 1.1 and each spec's related features (Expects,
Keep-Alives, etc), mucked with the 'stream write buffering', etc ... to no
avail.

After posting this yesterday, the developer attempted to recompile and rerun
using VS 2005 and .NET FW 2.0 - and early tests seems to indicate the issue
is resolved in the newer version of the Framework, as his code is exactly the
same as the code that fails in 1.1. Does anyone know if MS posts a list if
bug fixes that are incorporated into a framework release? I have to imagine
something was changed/fixed to now make this request successful.


"Joerg Jooss" wrote:

> Jeff Killberg wrote:
>
> > Thanks for the reply, Joerg.
> >
> > > Well, if you don't want Expectations at all, you can turn them off:
> > >
> > > // Do that before creating HttpWebRequest objects
> > > ServicePointManager.Expect100Continue = false;
> > I'll give a little bit more context - I am currently turning off
> > Expectations (using the same code as you show below) as well as
> > forcing the 1.0 protocol in my request.
>
> You shouldn't need to do both -- Expectations only exist in HTTP 1.1.
> Actually, I would never downgrade to HTTP 1.0, due to the severe
> limitations regarding cache control, redirects, yada yada...
>
> >
> > > 10 secs seems a tad aggressive to me, but I'd expect that to be
> > > configurable for your web server -- it isn't?
> > I completely agree - but the server is housed in a hardware
> > appliance, and the vendor says that this value is non-configurable.
>
> But either of the solutions discussed above does the trick, don't they?
>
> Cheers,
> --
> http://www.joergjooss.de
> mailto:news-(E-Mail Removed)
>

Jeff Killberg wrote:

[...]
> After posting this yesterday, the developer attempted to recompile
> and rerun using VS 2005 and .NET FW 2.0 - and early tests seems to
> indicate the issue is resolved in the newer version of the Framework,
> as his code is exactly the same as the code that fails in 1.1. Does
> anyone know if MS posts a list if bug fixes that are incorporated
> into a framework release? I have to imagine something was
> changed/fixed to now make this request successful.

I'm not aware of a complete document, but there's
http://download.microsoft.com/downlo...-462b-b9ad-e6d
3cd6ad7c1/Breaking%20Changes%20Beta2%20to%20RTM.doc
and
http://msdn.microsoft.com/library/de.../en-us/dnnetde
p/html/listofbreakingchanges.asp

Cheers,
--
http://www.joergjooss.de
mailto:news-(E-Mail Removed)