Reading the stream from a web response causes timeouts

[Guest]

Hi,

Im writing an application that crawls a website. One problem Ive come across
is that sometimes, when reading the contents of a web response, the
application will hang and timeout. This occurs when I am trying to read the
stream returned from the GetResponseStream of the HttpWebResponse object.

The code I am using is detailed below. I used the ReadFully method that Jon
Skeet outlines on his website, just to be safe.
http://www.yoda.arachsys.com/csharp/readbinary.html

In the example below, I find that while I can download the file and view it
using IE, when running my app, it frequently starts timing out after reading
aprx 700kb - if anyone could suggest where I am going wrong it would be
greatly appreciated.

Thanks in advance,

Mark Fletcher


using System;
using System.IO;
using System.Net;

namespace WebRequest2
{
/// <summary>
/// Summary description for Class1.
/// </summary>
class Driver
{
/// <summary>
/// The main entry point for the application.
/// </summary>
[STAThread]
static void Main(string[] args)
{
Uri uri = new
Uri("http://www.hubbardone.com/pdf/peer_to_peer_11-24-03.pdf");

HttpWebRequest httpRequest = null;
httpRequest = (HttpWebRequest)WebRequest.Create(uri);
httpRequest.Timeout = 500000;
httpRequest.KeepAlive = true;

HttpWebResponse httpResponse = null;
Stream responseStream = null;
try
{
httpResponse = (HttpWebResponse)httpRequest.GetResponse();

responseStream = httpResponse.GetResponseStream();
byte[] buffer = Driver.ReadFully(responseStream,1024);

Console.WriteLine("Finished");

}
catch(IOException ioex)
{
Console.WriteLine(ioex.Message);
}
catch(Exception ex)
{
Console.WriteLine(ex.Message);
}
finally
{
responseStream.Close();
httpResponse.Close();
}

}

public static byte[] ReadFully (Stream stream, int initialLength)
{
// If we've been passed an unhelpful initial length, just
// use 32K.
if (initialLength < 1)
{
initialLength = 32768;
}

byte[] buffer = new byte[initialLength];
int read=0;

int chunk;
while ( (chunk = stream.Read(buffer, read, buffer.Length-read)) > 0)
{
read += chunk;

// If we've reached the end of our buffer, check to see if there's
// any more information
if (read == buffer.Length)
{
int nextByte = stream.ReadByte();
// End of stream? If so, we're done
if (nextByte==-1)
{
return buffer;
}

// Nope. Resize the buffer, put in the byte we've just
// read, and continue
byte[] newBuffer = new byte[buffer.Length*2];
Array.Copy(buffer, newBuffer, buffer.Length);
newBuffer[read]=(byte)nextByte;
buffer = newBuffer;
read++;
}

}
// Buffer is now too big. Shrink it.
byte[] ret = new byte[read];
Array.Copy(buffer, ret, read);
return ret;
}


}
}

 

[Guest]

Hi,

Ive just recently had the chance to come back and look at this problem. It
appears that when I request the aforementioned file, the server is returning
it in chunks... the code is 206 - Partial Content

So in my code below, before I get the response, I call

httpRequest.AddRange(0,1000000);

Yet even so, the response only returns the first 700000 bytes, and I know
that by checking the content length it is far greater (aprx 3mb). Can anyone
out there help and suggest where I am going wrong?

How to handle code 206's with HttpWebRequest and HttpWebResponse?

Thanks in Advance,

Mark Fletcher

 

[Guest]

Hi, just updating it yet, but has anyone had any luck with this?

:

<Much Snipped>

 

[Guest]

Hi,

Each active thread makes one Http Request and gets the response.

I do find that I have occasional problems, when reading the response stream.
I find that when calling Stream.Read() this sometimes times out with the
following exception:

Message: The operation has timed out
Stack Trace: at System.ConnectStream.Read(...)

Now it doesnt matter if I set HttpRequest.Timeout = 10000 or 100000, I still
occasionally get these timeouts. When I debug the application its like the
thread is in the process of reading bytes part of the way through and then
hangs.

I can provide code if requested, but I was wondering if anyone else has seen
symptoms like this, and if they could suggest a remedy.

Thanks,

Mark