downloading a single file using multiple threads

[Jon Skeet [C# MVP]]

Wait a minute I guess your purpose is one again to pick on me, right?

Not at all - and I don't recall picking on you particularly in the past
either

It's just that when talking about HTTP and FTP I've always been
considering what I think of as the natural way of downloading multiple
chunks in multiple threads - using multiple requests on multiple
connections.

You repeatedly say (correctly, as far as I'm aware) that it can't be
done on a single connection (with HTTP/FTP) but that's a straw man that
no-one's actually suggested - so why keep talking about it?

I don't even know whether the OP isn't simply talking about a file
server request using System.IO.

Indeed. It would be nice if the OP would return to the thread

 

[Willy Denoyette [MVP]]

Agreed, but I'm not sure anyone suggested that as an option.


How would you do overlapping get requests on a single connection, out
of interest? That's something I haven't come across (and it sounds
pretty horrendous). Keeping one connection alive for multiple requests,
sure - but I haven't come across overlapping requests on one
connection.

I didn't suggest that either, I said that AFAIK it's not possible, so it wouldn't make sense
to have multiple threads issuing parallel requests using range requests over the same
session.

That's reasonable. I wouldn't like to fix it to 1 though, which is what
it sounded like you were suggesting before.

Where did you see me suggesting this? All I said was (in a reply to Peter and not to the
OP!) that it makes little or no sense to have multiple threads requesting ranges (we are
talking about one single file, right), not over the same connection and not over multiple
connections either as there is the limit of 2 connections as per HTTP1.1.


Willy.

 

[Peter Duniho]

[...] I don't even know if the protocol allows you to issue new range
request when you have a range requests pending (on the same logical
connection).

I doubt it does. I haven't tried it myself, but I don't see anything in
the HTTP specification that allows for any parallelism. I don't even see
how it would work, given that the connection is for all intents and
purposes a single stream of bytes.

Agreed, but what's the advantage in a simple client server scenario?
With simple I mean a simple PC connected over a dedicated LAN to an HTTP
server.

I see no advantage at all, in that scenario. Or, alternatively, if for
some reason creating more connections to the server allows for more
bandwidth, then the correct solution in that scenario is actually to fix
the server so that it's not throttling that client on the dedicated
connection.

[...]

And of course if the HTTP server is configured to throttle the
transfer for each connection, it is often also configured to disallow
multiple connections from the same client IP address.

They better do ;-)

Well, as Jon points out there are situations in which that wouldn't be
desirable. But I agree that in the file download situation, where a
"download manager" would be used, a server that throttles but doesn't
check for multiple connections from a single client is only
half-implemented.

Agreed, however, I don't know if this discussion is of any help to the
OP, let's see if he will come back.

Yeah, no kidding. I think we kind of went off on a bit of a tangent...who
knows if any of this part of the thread in any way relates to his original
issue. Guess we'll find out (or not).

He didn't even bother to specific the protocol he was using. However,
assuming he's using HTTP, hopefully the reference to the specification
will point him in the right direction. Note that the original question
doesn't really even presume parallel connections; he's just asking about
resuming a download where it got interrupted previously (well, at least
that's the scenario that seems to make the most sense given his actual
question).

Pete

 

[Peter Duniho]

[...]
One client A --- one Server,
two client threads,
sharing the same socket connection,
thread 1 issues asynchronous range request x...
thread 2 issues asynchronous range request y...
server accepts request x and at the same time accepts request y,
starts to return data for x and y.
AFAIK above is not possible, that doesn't mean it *fails* visibly.
IMO the server will choose to handle request x and queue request y, that
means he will return the data for x before he starts to return data for
y.

I think we all agree that that particular scenario won't work, and/or
isn't useful (as you say, even if you have enabled reuse of the
connection, repeated requests for different ranges will simply result in
queuing of the replies).

The only way to get some parallel action is to have multiple connections.

[...]
Yep, but say you have 1Mbps to the US and 50Kbps to Afrika, is your
download manager clever enough to request all chunks from the US instead
of waiting for a chunk to arrive from Afrika, or is he clever enough to
cancel the request and re-issue the same request over the US connection.

Any "download manager" worthy of the name ought to do that. I'm sure
there are plenty of download managers NOT worthy of the name, but at the
very least I would expect a download manager to break the download into
small enough chunks that if one connection is considerably slower than
another, the faster connection can wind up with the bulk of the effort.

For example, downloading 1MB might be broken into 100K sections. As each
section completes, a new request is issued, meaning that while a slow
server spends a long time on a single 100K section, a faster server gets
to service the rest of the 100K sections.

A more sophisticated program may even be intelligent enough to notice one
connection is faster and start issuing larger requests on that connection,
to avoid wasting too much time on the latency for a single GET. An even
more sophisticated program might even have logic to do as you suggest, and
cancel the slower connection when it becomes apparent that re-issuing the
remaining data from that request on a faster connection will net an
improvement (presumably the data already retrieved would not be discarded).

It's clear that some amount of thought does need to go into implementing a
servicable "download manager" that actually improves the situation when
downloading files. But I've seen plenty of people just in this newsgroup
with the insight to be able to do this (I'd dare say the three of us
making so much noise in this thread are among them ), and I don't think
any of these issues in and of themselves suggest that there's no benefit
to parallel downloads of different sections of the same file.

This requires permanent real-time monitoring of the transfer rates,
something that can't reliably be done at the application layer,

Why do you say that real-time monitoring of the transfer rates can't be
reliably done at the application layer? I agree that you can't get
detailed information about the exact rate of transfer, and especially of
why the speed is fast or slow (a fast connection might have low throughput
due to errors, for example). But at the granularity required to monitor a
download and automatically compensate for slow and fast connections,
sufficient for a user to notice an improvement in download performance,
I'd say at the application layer you can easily get sufficient information.

Given that a download manager is most useful for downloads that take tens
of minutes or even more, just following the averages is plenty sufficient
to significantly improve performance. And of course, the bulk of the
improvement would come simply from having parallel connections, which
doesn't require any monitoring of throughput anyway (yes, with such a
simple approach there would be cases where things didn't get better, or
even got worse, but most of the time there would be a net improvement).

that's what I meant when I said you really need sophisticated software
and be sure to measure (as always) the real benefits, I guess
(expensive) download managers can do it while other pretend they can,
big difference!.

Assuming there's a such thng as an "expensive download manager" (I
wouldn't know, not being in the market for one), then yes...I'd hope they
would at least go to as much trouble as described above. But I think all
that can easily be handled at the application layer.

Pete

 

[Willy Denoyette [MVP]]

Not at all - and I don't recall picking on you particularly in the past
either

Yes, you did, I prefer not to go public on this.

It's just that when talking about HTTP and FTP I've always been
considering what I think of as the natural way of downloading multiple
chunks in multiple threads - using multiple requests on multiple
connections.

You repeatedly say (correctly, as far as I'm aware) that it can't be
done on a single connection (with HTTP/FTP) but that's a straw man that
no-one's actually suggested - so why keep talking about it?

I guess that Peter suggested this (excuse Peter if I got this wrong) by saying

See, for example, the "Range" field in HTTP.

http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35.1

(I'm guessing that the above link may actually answer the question for the
OP)

well, having multiple threads without the possibility to have overlapped range requests per
connection makes no sense, having multiple threads requesting chunks over 2 connections
(HTTP1.1) makes little sense IMO either, unless you have measured the real benefits.

Indeed. It would be nice if the OP would return to the thread

That's the real problem these day's, they come up with a vague question they get some
confusing answers (our mistakes?), finally they get side tracked (by us) and they never come
back, they are scared off, really.

Willy.

 

[Jon Skeet [C# MVP]]

I didn't suggest that either, I said that AFAIK it's not possible, so
it wouldn't make sense to have multiple threads issuing parallel
requests using range requests over the same session.

I know you didn't suggest it - but you keep coming back to it to say it
won't work, when I don't think anyone's said that it *would* work, or
that anyone should try it.

Where did you see me suggesting this? All I said was (in a reply to
Peter and not to the OP!) that it makes little or no sense to have
multiple threads requesting ranges (we are talking about one single
file, right), not over the same connection and not over multiple
connections either as there is the limit of 2 connections as per
HTTP1.1.

Here (with Peter's text indented with the single angle bracket)

And of course if the HTTP server is configured to throttle the
transfer for each connection, it is often also configured to disallow
multiple connections from the same client IP address.

They better do ;-)
</quote>

That suggests that you believe servers with transfer throttling should
disallow multiple connections from the same IP address. To me, 2
connections counts as "multiple". I believe it's perfectly reasonable
to have connection throttling and yet still allow 2 connections from a
single client IP address.

 

[Jon Skeet [C# MVP]]

He didn't even bother to specific the protocol he was using. However,
assuming he's using HTTP, hopefully the reference to the specification
will point him in the right direction. Note that the original question
doesn't really even presume parallel connections; he's just asking about
resuming a download where it got interrupted previously (well, at least
that's the scenario that seems to make the most sense given his actual
question).

It doesn't presume parallel connections, but it *does* talk about
multiple threads which assumes parallelisation in some form or other. I
certainly *hope* he also means parallel connections

 

[Peter Duniho]

It doesn't presume parallel connections, but it *does* talk about
multiple threads which assumes parallelisation in some form or other. I
certainly *hope* he also means parallel connections

I hope so too. But you've been here WAY long enough to know that you get
all kinds of whacky posts here sometimes. There's no way to know for sure
that he doesn't intend to use a single connection from multiple threads.

Too often, I make the incorrect assumption that the person posting knows
the difference between a good idea and a bad one. I always hope
that's not the case, but until the person clarifies, it's difficult or
impossible to know for sure.

Pete

 

[Jon Skeet [C# MVP]]

Yes, you did, I prefer not to go public on this.

Well, my sincere apologies if it felt like that, whichever thread you
mean. I'm certainly not bearing a grudge against you or anything like
that.

I guess that Peter suggested this (excuse Peter if I got this wrong) by saying

See, for example, the "Range" field in HTTP.

http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35.1

(I'm guessing that the above link may actually answer the question for the
OP)

well, having multiple threads without the possibility to have
overlapped range requests per connection makes no sense having
multiple threads requesting chunks over 2 connections (HTTP1.1) makes
little sense IMO either, unless you have measured the real benefits.

Presumably the OP hasn't measured the benefits as he doesn't know how
to do it yet (and of course we still don't know if it's HTTP!) but I
certainly didn't see any implication of connection-sharing in Peter's
post. I suspect this conversation would have been quite different in
the flesh, with non-verbal signals to help.

That's the real problem these day's, they come up with a vague
question they get some confusing answers (our mistakes?), finally
they get side tracked (by us) and they never come back, they are
scared off, really.

Or, alternatively, come back saying "The problem is solved" without
leaving anything for future readers in terms of how the problem was
solved, or what the problem even was...

 

[Peter Duniho]

[...]

Not at all - and I don't recall picking on you particularly in the past
either

Yes, you did, I prefer not to go public on this.

For what it's worth, I didn't get the impression that anyone was picking
on anyone else.

When such an impression is received by someone, it's also been my
experience that it is almost always actually a case of one person simply
trying to make sure they have the facts straight (and in the process
drilling down into more detail than someone else thinks is necessary or
warranted), or a case of two people who think they are talking about the
same thing turning out to not be talking about the same thing, or both.

Honestly, I have seen precious little of what I'd call "picking on" in
this newsgroup. It's one of the reasons it's one of the few newsgroups I
still bother to read.

That said...

I guess that Peter suggested this (excuse Peter if I got this wrong) by
saying

See, for example, the "Range" field in HTTP.

http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35.1

You are excused.

If you'll look, you'll note that I posted that simply in reply to your
statement about whether it was possible to request a file fragment via
HTTP or FTP. The context had nothing to do (at the time) with numbers of
connections, sessions, etc. It was just a question of whether those
protocols had any way to specify some subset during a download.

In particular, here is the statement to which I replied (and quoted in my
post):

"How do you indicate what chunk of the file you want when say using FTP?
As far as I know this is not part of the FTP neither of HTTP protocol"

Nothing in there about connections or sessions. Just the question of
indicating what chunk of a file you want when using FTP or HTTP.

well, having multiple threads without the possibility to have overlapped
range requests per connection makes no sense, having multiple threads
requesting chunks over 2 connections (HTTP1.1) makes little sense IMO
either, unless you have measured the real benefits.

Well, as has been adequately described already I think, there are
definitely scenarios in which there is a measurable benefit.

That's the real problem these day's, they come up with a vague question
they get some confusing answers (our mistakes?), finally they get side
tracked (by us) and they never come back, they are scared off, really.

Well, hopefully the worthy ones recognize the ambiguity in their question
and post a clarification.

But I'd agree that it's our fault when we answer a question before we
really understand it. Hopefully we haven't done too much of that here.

Pete

 

[Willy Denoyette [MVP]]

[...]
One client A --- one Server,
two client threads,
sharing the same socket connection,
thread 1 issues asynchronous range request x...
thread 2 issues asynchronous range request y...
server accepts request x and at the same time accepts request y, starts to return
data for x and y.
AFAIK above is not possible, that doesn't mean it *fails* visibly.
IMO the server will choose to handle request x and queue request y, that means he will
return the data for x before he starts to return data for y.

I think we all agree that that particular scenario won't work, and/or isn't useful (as
you say, even if you have enabled reuse of the connection, repeated requests for
different ranges will simply result in queuing of the replies).

The only way to get some parallel action is to have multiple connections.

Yep, that's the basic requirement.

[...]
Yep, but say you have 1Mbps to the US and 50Kbps to Afrika, is your download manager
clever enough to request all chunks from the US instead of waiting for a chunk to arrive
from Afrika, or is he clever enough to cancel the request and re-issue the same request
over the US connection.

Any "download manager" worthy of the name ought to do that. I'm sure there are plenty of
download managers NOT worthy of the name, but at the very least I would expect a download
manager to break the download into small enough chunks that if one connection is
considerably slower than another, the faster connection can wind up with the bulk of the
effort.

Agreed, and all say they support HTTP, but what does this means, really. No one measures the
throughput benefits in real time, as it's really hard to measure (over the internet). The
same is true for those magical memory managers and memory defragmenters, they all pretend to
optimize memory access, but once you really start measuring, you'll notice they are
worthless.

For example, downloading 1MB might be broken into 100K sections. As each section
completes, a new request is issued, meaning that while a slow server spends a long time
on a single 100K section, a faster server gets to service the rest of the 100K sections.

A more sophisticated program may even be intelligent enough to notice one connection is
faster and start issuing larger requests on that connection, to avoid wasting too much
time on the latency for a single GET. An even more sophisticated program might even have
logic to do as you suggest, and cancel the slower connection when it becomes apparent
that re-issuing the remaining data from that request on a faster connection will net an
improvement (presumably the data already retrieved would not be discarded).

It's clear that some amount of thought does need to go into implementing a servicable
"download manager" that actually improves the situation when downloading files. But I've
seen plenty of people just in this newsgroup with the insight to be able to do this (I'd
dare say the three of us making so much noise in this thread are among them ), and I
don't think any of these issues in and of themselves suggest that there's no benefit to
parallel downloads of different sections of the same file.


Why do you say that real-time monitoring of the transfer rates can't be reliably done at
the application layer? I agree that you can't get detailed information about the exact
rate of transfer, and especially of why the speed is fast or slow (a fast connection
might have low throughput due to errors, for example). But at the granularity required
to monitor a download and automatically compensate for slow and fast connections,
sufficient for a user to notice an improvement in download performance, I'd say at the
application layer you can easily get sufficient information.

Well, you can't get enough detailed (or the right) information about the transfer rates
based on a per socket connection in user space, you can't even get at the throughput at the
application (port) level. So you need some help of a NIC device driver of filter driver to
coordinate with the application. Something like this is done with "load balancing" feature
in a farm or cluster server, but this is at the server side, I'm talking about the client
here.
Sure, you can measure at a reasonably level of precision using simple client PC, running a
single application you are monitoring in isolation, but this picture changes when you have a
server with multiple NIC's, connected over segmented LAN's, that serves as a download client
(such beasts do exist). Here it's important to have nearly real-time performance figures
about the throughputs per application per connection (both physical and logical), such that
you can effectively balance the loads over the NIS's and as such over the segments.
I know, I'm going a bit far off to illustrate the point, but it does exists as a product
(what I call expensive) and I know who's using such things.

Given that a download manager is most useful for downloads that take tens of minutes or
even more, just following the averages is plenty sufficient to significantly improve
performance. And of course, the bulk of the improvement would come simply from having
parallel connections, which doesn't require any monitoring of throughput anyway (yes,
with such a simple approach there would be cases where things didn't get better, or even
got worse, but most of the time there would be a net improvement).


Assuming there's a such thng as an "expensive download manager" (I wouldn't know, not
being in the market for one), then yes...I'd hope they would at least go to as much
trouble as described above. But I think all that can easily be handled at the
application layer.

They exist, but you can't buy them in a PC store ;-).

Willy.

 

[Willy Denoyette [MVP]]

I know you didn't suggest it - but you keep coming back to it to say it
won't work, when I don't think anyone's said that it *would* work, or
that anyone should try it.

Well, I'm not coming back, I'm just asking how one could link " multiple threads " (that is
2 or more) to "multiple range requests" given you have (as per HTTP 1.1) only two
connections possible. But, promissed, I'm not coming back on this, ever ;-)

Here (with Peter's text indented with the single angle bracket)



They better do ;-)
</quote>

Well, guess I was thinking "they better restrict "instead of "disallow", my bad, sorry.

That suggests that you believe servers with transfer throttling should
disallow multiple connections from the same IP address. To me, 2
connections counts as "multiple". I believe it's perfectly reasonable
to have connection throttling and yet still allow 2 connections from a
single client IP address.

Sorry, if I made you think that the server should refuse multiple connections, I'm
suggesting they follow the HTTP 1.1 recommendations, as per default, and that they allow you
configure that number for dedicated purposes. Note also that a server (here IIS) doesn't
enforce this rule, it relies on the client to obey HTTP1. If IIS gets' multiple connection
requests (>2) from the same client (IP address) he won't refuse a connection, he will
simply handle the first x request and enqueue the others depending on a number of
heuristics, the pending request are subject to timeout restrictions at the client. But
assuming that he will handle tens or hundreds of parallel requests from the same client is
simply wrong,

Willy.

 

[Jon Skeet [C# MVP]]

Well, I'm not coming back, I'm just asking how one could link "
multiple threads " (that is 2 or more) to "multiple range requests"
given you have (as per HTTP 1.1) only two connections possible. But,
promissed, I'm not coming back on this, ever ;-)

2 per server (or more if you find you can get away with it) - as I
said, there's the possibility of using mirrors too.

Admittedly you have to be pretty confident in the mirrors - I'd
certainly want an md5 checksum for the combined file afterwards

 

[Willy Denoyette [MVP]]

Well, my sincere apologies if it felt like that, whichever thread you
mean. I'm certainly not bearing a grudge against you or anything like
that.

No big deal, I guess I'm a little tired, note also that English is not my first (nor my
second) language, I'm not sure whether my "interpretation" is always the right one :-(

Presumably the OP hasn't measured the benefits as he doesn't know how
to do it yet (and of course we still don't know if it's HTTP!) but I
certainly didn't see any implication of connection-sharing in Peter's
post. I suspect this conversation would have been quite different in
the flesh, with non-verbal signals to help.


Or, alternatively, come back saying "The problem is solved" without
leaving anything for future readers in terms of how the problem was
solved, or what the problem even was...

Yes, that's even worse, but I know some get scared off, they start sending private mails (as
per proper experience) because they don't want (or can't) enter the discussion. Maybe we
(MVP's especially) should pay more attention to stay OT and don't get ourselves
side-tracked, not that it could not lead to a good discussion (like this), but it won't
help the OP I'm afraid.

Willy.

 

[Willy Denoyette [MVP]]

[...]

Not at all - and I don't recall picking on you particularly in the past
either

Yes, you did, I prefer not to go public on this.

For what it's worth, I didn't get the impression that anyone was picking on anyone else.

When such an impression is received by someone, it's also been my experience that it is
almost always actually a case of one person simply trying to make sure they have the
facts straight (and in the process drilling down into more detail than someone else
thinks is necessary or warranted), or a case of two people who think they are talking
about the same thing turning out to not be talking about the same thing, or both.

Honestly, I have seen precious little of what I'd call "picking on" in this newsgroup.
It's one of the reasons it's one of the few newsgroups I still bother to read.

That said...

I guess that Peter suggested this (excuse Peter if I got this wrong) by saying

See, for example, the "Range" field in HTTP.

http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35.1

You are excused.

Thanks

If you'll look, you'll note that I posted that simply in reply to your statement about
whether it was possible to request a file fragment via HTTP or FTP. The context had
nothing to do (at the time) with numbers of connections, sessions, etc. It was just a
question of whether those protocols had any way to specify some subset during a download.

In particular, here is the statement to which I replied (and quoted in my post):

"How do you indicate what chunk of the file you want when say using FTP? As far as I know
this is not part of the FTP neither of HTTP protocol"

Nothing in there about connections or sessions. Just the question of indicating what
chunk of a file you want when using FTP or HTTP.


Well, as has been adequately described already I think, there are definitely scenarios in
which there is a measurable benefit.

Well, I was focused on a different scenario - multiple range requests over a single
connection - and my mistake was that you were thinking the same way.

Well, hopefully the worthy ones recognize the ambiguity in their question and post a
clarification.

But I'd agree that it's our fault when we answer a question before we really understand
it. Hopefully we haven't done too much of that here.

Yep, I guess we are trying too hard to help, instead of asking further questions so we get a
better idea about the issue at hand.

Willy.

 

[Willy Denoyette [MVP]]

2 per server (or more if you find you can get away with it) - as I
said, there's the possibility of using mirrors too.

Admittedly you have to be pretty confident in the mirrors - I'd
certainly want an md5 checksum for the combined file afterwards

Sure, so do I, I remember I had to download the latest Windows SDK (not from msdn
subscribers though) three of four of times from a mirror before I got an installable copy.
The problem however is that you need to download before you can check, and you need to
restart if the check fails, this is frustrating especially when the download is a single
image of couple of GB.

Willy.