Advice on Handling Large File Download via HTTP

[Mark Olbert]

I'm writing a C# app which downloads large files via http from a server. These files are often in excess of a gigabyte.

I am trying to figure out the best way to download these kinds of large files in .NET. While I've successefully downloaded test
files using WebClient, I don't think that's going to be a viable final solution because I need to be able process the data stream
before writing it to the local disk (basically, I need to strip off certain bytes). I haven't found a WebClient method which allows
me to "intercept" the downloaded data stream to process it.

I've played around with WebRequest and WebResponse, but am running into a situation where no more than the first 256K bytes of data
is readable. In other words, if I do something like this (pseudo code):

WebResponse response = request.GetResponse();
Stream stream = response.GetResponseStream();

while( not enough bytes read yet )
{
stream.Read(buffer...)
localFileStream.Write(buffer...)
}

only the first 256K bytes are read from the stream. After that, the stream.Read() has a return value of 0.

So...what advice do folks have on how I should move forward?

- Mark

 

[Mark Olbert]

One correction: the "256K byte limit" problem was self-inflicted (some old logic I forgot to remove).

But I'm still interested in advice on downloading large files via http.

- Mark

 

[Joerg Jooss]

Thus wrote Mark,

I'm writing a C# app which downloads large files via http from a
server. These files are often in excess of a gigabyte.

I am trying to figure out the best way to download these kinds of
large files in .NET. While I've successefully downloaded test

files using WebClient, I don't think that's going to be a viable final
solution because I need to be able process the data stream

before writing it to the local disk (basically, I need to strip off
certain bytes). I haven't found a WebClient method which allows

me to "intercept" the downloaded data stream to process it.

When you use WebClient.OpenStream(), you can process the reponse stream directly.


But for GB sized download, I'd rather think about something more advanced
than plain HTTP, like BITS for example.

Cheers,

 

[Yuan Ren[MSFT]]

Hi Mark,

Thanks for posting!

I'm glad to hear the issue has been resolved:

In addition, there are many samples from third party demonstrate how to
approach this. I suggest you follow two articles below:
"Cool C# File Downloader":
http://www.codeproject.com/csharp/CoolDownloader.asp

"Threaded WebDownload class with Progress Call-backs":
http://www.codeproject.com/csharp/webdownload.asp

They are appropriated at the current stage. Thanks for your understanding!

Regards,

Yuan Ren [MSFT]
Microsoft Online Support