Disabling chunking in HTTP 1.1

[joseph_gallagher]

Hi,

I'm not sure if this is the right place to post this but when
searching for the exception I got this group came up the most.

I'm making a HttpWebRequest to get data from a web page but am getting
the following error

System.Net.WebException: The server committed a protocol violation.
Section=ResponseBody Detail=Response chunk format is invalid
at System.Net.HttpWebRequest.GetResponse()

I've looked in the groups and the recomended fix for this error is to
set

<system.net>
<settings>
<httpWebRequest useUnsafeHeaderParsing="true" />
</settings>
</system.net>

in the App.config but this isnt working for me, playing around I find
that if I set the version in the HttpWebRequest to 1.0 like so

HttpWebRequest hwr = (HttpWebRequest)
HttpWebRequest.Create(url.Text);
hwr.ProtocolVersion = new System.Version(1,0);

then everything runs fine, running a packet sniffer I see this is
becuase I'm not recieving the data chunked, is there a header I can
set so I can carry on using 1.1 but not get chunked data as this is
obviously whats causing the problem and I dont really want to use http
1.0.

Failing that does anyone know what is causing this problem?

 

[Joerg Jooss]

Thus wrote (e-mail address removed),

Hi,

I'm not sure if this is the right place to post this but when
searching for the exception I got this group came up the most.

It's the right place

I'm making a HttpWebRequest to get data from a web page but am getting
the following error

System.Net.WebException: The server committed a protocol violation.
Section=ResponseBody Detail=Response chunk format is invalid
at System.Net.HttpWebRequest.GetResponse()
I've looked in the groups and the recomended fix for this error is to
set

<system.net>
<settings>
<httpWebRequest useUnsafeHeaderParsing="true" />
</settings>
</system.net>
in the App.config but this isnt working for me, playing around I find
that if I set the version in the HttpWebRequest to 1.0 like so

That would be a severe bug on the server-side. Is this a public site? Or
is there a web proxy in between?

HttpWebRequest hwr = (HttpWebRequest)
HttpWebRequest.Create(url.Text);
hwr.ProtocolVersion = new System.Version(1,0);
then everything runs fine, running a packet sniffer I see this is
becuase I'm not recieving the data chunked, is there a header I can
set so I can carry on using 1.1 but not get chunked data as this is
obviously whats causing the problem and I dont really want to use http
1.0.

Understandable. But it's the server's decision whether to use chunking or
not. A HTTP 1.1 client must support chunking, so there's no way to avoid
chunking without reverting to HTTP 1.0.

Failing that does anyone know what is causing this problem?

Not without having a look at the actual traffic.

Cheers,

 

[joseph_gallagher]

Thanks for the reply Joerg, it is a public server and I'm not going
through any proxies so it is strange, its also strange that it only
happens randomly but quiet frequently, around 50% of the time, some
sample code that calls the page in question is

[STAThread]
public static void Main() {
string url = "http://site.sports.betfair.com/menu/
LoadMenuNodesAction.do?
sReturnPath=parent.frames['menu']&method=getMenuEvents&menuNodeId=11589568&strArrayName=allSkeletonArray&iParentID=11589568&layerName=allMarketsTreeContainer&strMenuPathArrayName=allPathArray&menuPathLayer=menuParents1&locale=en_GB";
for(int i = 0 ; i < 10 ; i++) {
HttpWebRequest hwr = (HttpWebRequest)
HttpWebRequest.Create(url);
try {
using(WebResponse wr = hwr.GetResponse()) {
using(Stream s = wr.GetResponseStream()) {
using(StreamReader sr = new
StreamReader(s)) {
Console.WriteLine(sr.ReadToEnd());
}
}
}
} catch(WebException we) {
Console.WriteLine(we.ToString());
}
Console.ReadLine();
}
}

Running a packet sniffer I can see that the correct text is been
returned but I'm not sure if it is been chunked correctly, its also
strange that the url works 100% in IE.

Thanks for any help, if you dont have time to look at it further dont
worry about it, usuing 1.0 is an acceptable alternative as its a call
that is only used occasionaly, I'm more just curious than anything
else.

Joe.

 

[Alan J. McFarlane]

Thanks for the reply Joerg, it is a public server and I'm not going
through any proxies so it is strange, its also strange that it only
happens randomly but quiet frequently, around 50% of the time, some
sample code that calls the page in question is
[/QUOTE]
Seems to work ok always for me, at least I've not repro'd any error in
the number of times that I've run it... Maybe someone else will find
different. There's no chance there's a transparent proxy in your path,
see if there's an Via header in the response for instance.

If you've a sniffer trace of case where it fails can you upload that
somewhere and we can have a look to see if there's something obviously
wrong with the response.

The format of chunked is relatively simple { n n n CR LF {data of size
nnn} CR LF } for each chunk, with the end marked with a zero length
chunk, and the length 'nnn' is an ascii formatted number
so the first trace I have has this response:

HTTP/1.1 200 ....
....
Flags: ... BRefresh=onCRLF
CRLF
65CRLF
{65 bytes of content}CRLF
289CRLF
{289 bytes of content}CRLF
0CRLF
CRLF

Seems ok -- should do as it was a successful download. (Not too sure
about the very last CRLF, but...)

 

[Joerg Jooss]

Thus wrote (e-mail address removed),

Thanks for the reply Joerg, it is a public server and I'm not going
through any proxies so it is strange, its also strange that it only
happens randomly but quiet frequently, around 50% of the time, some
sample code that calls the page in question is

I can see some pretty strange artifacts here when enabling unsafe header
parsing -- it resembles a split HTTP response.

[STAThread]
public static void Main() {
string url = "http://site.sports.betfair.com/menu/
LoadMenuNodesAction.do?
sReturnPath=parent.frames['menu']&method=getMenuEvents&menuNodeId=1158
9568&strArrayName=allSkeletonArray&iParentID=11589568&layerName=allMar
ketsTreeContainer&strMenuPathArrayName=allPathArray&menuPathLayer=menu
Parents1&locale=en_GB";
for(int i = 0 ; i < 10 ; i++) {
HttpWebRequest hwr = (HttpWebRequest)
HttpWebRequest.Create(url);
try {
using(WebResponse wr = hwr.GetResponse()) {
using(Stream s = wr.GetResponseStream()) {
using(StreamReader sr = new
StreamReader(s)) {
Console.WriteLine(sr.ReadToEnd());
}
}
}
} catch(WebException we) {
Console.WriteLine(we.ToString());
}
Console.ReadLine();
}
}
Running a packet sniffer I can see that the correct text is been
returned but I'm not sure if it is been chunked correctly, its also
strange that the url works 100% in IE.

IE is much more lenient regarding HTTP issues than System.Net.

Thanks for any help, if you dont have time to look at it further dont
worry about it, usuing 1.0 is an acceptable alternative as its a call
that is only used occasionaly, I'm more just curious than anything
else.

Try to change your code as follows:

Instead of decoding the response using a StreamReader, simply dump the response
stream to a MemoryStream. After you've read the entire response, decode it
using System.Text.Encoding.UTF8.GetString(byte[]).

Cheers,