Download URL Up to XXX Bytes

[Jonathan Wood]

I'm writing a desktop C# app that downloads files from the Web. However, I'd
like to limit the download to one MB or so.

Does anyone know of any examples to do this? I found simple examples using
either WebClient.DownloadString() or StreamReader.ReadToEnd() but they don't
support reading only part of the file, and I found other examples that
support reading part of the file but do not work with strings and involve
methods not supported by a stream coming over the Web.

Thanks.

Jonathan

 

[Jonathan Wood]

Just read from the stream directly (see NetworkStream), counting bytes as
you go.

I'll look for NetworkStream. But, it's the bytes and encoding issues that
were one of the issues I was referring to whereas the routines to read the
entire file deal with strings, which is what I'll ultimately need.

Ideally, you'd look at the web response and check the length of the
content before you even bother reading. That way, you only actually wind
up reading if the content is within the limit, or the length isn't
provided.

Sounds good to me. Perhaps someone will run across this thread who knows how
to accomplish this.

Thanks.

Jonathan

 

[Jonathan Wood]

Not really sure what you mean, but the Encoding/Encoder classes in
System.Text are useful for dealing with encoding. You need to work at the
byte level anyway, if you expect to monitor actual bytes. StreamReader
(for example) only gives you information about characters read, not bytes.

Characters is what I want. Here's what I'm doing now:

public static string DownloadUrl(string url, string userAgent)
{
WebClient client = new WebClient();
using (Stream strm = client.OpenRead(url))
{
StreamReader sr = new StreamReader(strm);
return sr.ReadToEnd();
}
}

Or even:

public static string DownloadUrl(string url, string userAgent)
{
WebClient client = new WebClient();
return client.DownloadString(url);
}

Note that I don't need to deal with bytes, encoding or much of anything else
for that matter. That's good because I'm very new to stuff like character
encoding and converting between strings and bytes.

I'd simply like to stop reading once the number of bytes OR characters is a
certain amount.

I searched on NetworkStream but wasn't able to see what it is doing or find
anything that resembled a complete sample that's doing what I'm trying to
do.

I think you must mean "who will write my code for me"? At least one
person has already run across the thread who "knows how to accomplish
this" (i.e. me), but the docs are already pretty clear on how to find out
the content length (i.e. WebResponse.ContentLength property), so the rest
should be reasonably straightforward. I don't know what else I'd add.

I happen to think much of the docs are piss-poor. I was just hoping for a
little sample of downloading a file without downloading the entire thing.

Jonathan

 

[Jonathan Wood]

That's what I figured.

So, it sounds then like you think the purpose of these newsgroups is just to
tell folks to RTFM.

Very well, I'm not new to development in general, and I stand by my
statement that the official docs leave much to be desired.

Jonathan

 

[Arne Vajhøj]

I'm writing a desktop C# app that downloads files from the Web. However,
I'd like to limit the download to one MB or so.

Does anyone know of any examples to do this? I found simple examples
using either WebClient.DownloadString() or StreamReader.ReadToEnd() but
they don't support reading only part of the file, and I found other
examples that support reading part of the file but do not work with
strings and involve methods not supported by a stream coming over the Web.

HttpWebRequest req =
(HttpWebRequest)WebRequest.Create("http://localhost/tst.txt");
req.AddRange(0, xxx);
HttpWebResponse resp = (HttpWebResponse)req.GetResponse();
....

will work with most web servers including IIS and Apache.

(you can also add the header to WebClient Headers, but for
this type of control HttpWebRequest seems more appropriate)

Arne

 

[Jonathan Wood]

That's very interesting, thanks. However, it's definitely a concern if some
servers won't support it.

I'll look into this a bit more. Thanks again.

 

[Arne Vajhøj]

That's very interesting, thanks. However, it's definitely a concern if
some servers won't support it.

I'll look into this a bit more.

I would expect practically all recent general purpose web servers
to support it (and Apache and IIS do count for approx. 90% of the market).

If you need to download from special web servers, then all
chances are not good. If I were to implement a web server
using System.Net.HttpListener for some app management, then
I would not implement byte range.

Arne

 

[Jonathan Wood]

Right.

The main point is that I'd like the code to work with many servers--not just
this one or that one. But definitely worth exploring.