downloading a single file using multiple threads

[keerthyragavendran]

hi

i'm downloading a single file using multiple threads...
how can i specify a particular range of bytes alone from a single
large file... for example say if i need only bytes ranging from
500000 to 3200000 of a file whose size is say 20MB...
how do i request a download which starts directly at 500000th byte...
thank u
cheers

 

[Jon Skeet [C# MVP]]

i'm downloading a single file using multiple threads...
how can i specify a particular range of bytes alone from a single
large file... for example say if i need only bytes ranging from
500000 to 3200000 of a file whose size is say 20MB...
how do i request a download which starts directly at 500000th byte...

That completely depends on what protocol you're using. Could you give
us more information?

 

[Willy Denoyette [MVP]]

hi

i'm downloading a single file using multiple threads...
how can i specify a particular range of bytes alone from a single
large file... for example say if i need only bytes ranging from
500000 to 3200000 of a file whose size is say 20MB...
how do i request a download which starts directly at 500000th byte...
thank u
cheers

Simple answer: you can't *download* a file in chunks, unless you implement a client/server
protocol for this.
If by *download* you mean *reading* a file from a "fileserver" and saving a copy to the
local filesystem, then you can use System.IO namespace classes
The question is - what makes you think you need multiple threads to *download* a file? If
you think you can speed-up the download by doing this, then you are wrong. The bottleneck
will always be the network, so if you download the file in one chunk, you'll get the maximum
throughput, introducing multiple threads will actually slowdown the whole process.

Willy.

 

[Jon Skeet [C# MVP]]

Simple answer: you can't *download* a file in chunks, unless you implement a client/server
protocol for this.

Note that there already protocols which *do* support this, including
FTP and HTTP, both optionally as far as the server is concerned.

The question is - what makes you think you need multiple threads to *download* a file? If
you think you can speed-up the download by doing this, then you are wrong. The bottleneck
will always be the network, so if you download the file in one chunk, you'll get the maximum
throughput, introducing multiple threads will actually slowdown the whole process.

That depends - some servers may throttle per connection, at which
point it may make sense to have multiple connections (although
somewhat naughty). Also, in a more advanced way, if the same file is
available through multiple mirrors, it may make sense to get different
bits from different mirrors.

This is a fairly common thing to do - a lot of web "download managers"
do it.

Jon

 

[Willy Denoyette [MVP]]

Note that there already protocols which *do* support this, including
FTP and HTTP, both optionally as far as the server is concerned.

A *single file* download from an FTP or HTTP server? How do you indicate what chunk of the
file you want when say using FTP? As far as I know this is not part of the FTP neither of
HTTP protocol.

That depends - some servers may throttle per connection, at which
point it may make sense to have multiple connections (although
somewhat naughty). Also, in a more advanced way, if the same file is
available through multiple mirrors, it may make sense to get different
bits from different mirrors.

This is a fairly common thing to do - a lot of web "download managers"
do it.

Per file? I mean, retrieve a *single file* from multiple servers in chunks? Say the first 10
MB's from server A the next 2MB from server B, the next 3 MB from server C? Never heard of
something like this.
All I know that there are download managers that download file A from server A, file B from
server B etc.. in case of multi-file downloads.

Willy.

 

[Peter Duniho]

A *single file* download from an FTP or HTTP server? How do you indicate
what chunk of the file you want when say using FTP? As far as I know
this is not part of the FTP neither of HTTP protocol.

See, for example, the "Range" field in HTTP.

http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35.1

(I'm guessing that the above link may actually answer the question for the
OP)

Per file? I mean, retrieve a *single file* from multiple servers in
chunks? Say the first 10 MB's from server A the next 2MB from server B,
the next 3 MB from server C? Never heard of something like this.

Well, it does happen. Most larger servers monitor the client IP address
and refuse additional connections, and with smaller servers doing this
sort of thing is considered anti-social, since it intentionally bypasses
per-client throttling that has been set up. But it's true that there are
"download manager" programs that do exactly what Jon is referring to.

All I know that there are download managers that download file A from
server A, file B from server B etc.. in case of multi-file downloads.

Note that there's nothing to stop a download manager from retrieving
different parts of the same file from multiple servers as well. Assuming
an identical file stored on various mirrors, it doesn't matter which
mirror a given part of the file comes from.

Pete

 

[Willy Denoyette [MVP]]

See, for example, the "Range" field in HTTP.

http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35.1

I know that HTTP 1.1 supports range requests, but this is not exactly my point.
The multi part requests in HTTP1.1 are meant to request (for very specic application
purposes) a single part or multiple parts in a single request, but you can't (AFAIK)
requests multiple parts in parallel from multiple client threads.

(I'm guessing that the above link may actually answer the question for the OP)


Well, it does happen. Most larger servers monitor the client IP address and refuse
additional connections, and with smaller servers doing this sort of thing is considered
anti-social, since it intentionally bypasses per-client throttling that has been set up.
But it's true that there are "download manager" programs that do exactly what Jon is
referring to.


Note that there's nothing to stop a download manager from retrieving different parts of
the same file from multiple servers as well. Assuming an identical file stored on
various mirrors, it doesn't matter which mirror a given part of the file comes from.

That's true, but these will use dedicated protocols don't they? The clients also should have
multiple NIC's installed connected over segmented LAN's and/or routers to take some speed
advantage of the parallelism.

Willy.

 

[Peter Duniho]

I know that HTTP 1.1 supports range requests, but this is not exactly my
point.
The multi part requests in HTTP1.1 are meant to request (for very specic
application purposes) a single part or multiple parts in a single
request, but you can't (AFAIK) requests multiple parts in parallel from
multiple client threads.

Well, I've never actually tried it myself. So I will refrain from
claiming that I know firsthand that this generally works. However, the
so-called "download managers" all claim to work with HTTP servers. I have
no reason to doubt them, and based on what I know about HTTP (which is an
extremely simple protocol) it seems likely.

There is no theoretical reason why it wouldn't work. There's nothing
about HTTP that requires servers to restrict their communications to a
given client to a single connection, and there's nothing about HTTP that
stipulates that an HTTP server needs to coordinate communications on
independent connections. If on one connection the client asks for the
first megabyte and on a second connection the same client asks for the
second megabyte, then if the server is capable of servicing both requests
at the same time, there's no reason the client can't wind up receiving
both the first and second megabytes in parallel.

Jon has already outlined the scenarios in which doing so would be
helpful. You are absolutely correct that in many cases, the HTTP server
is already providing data as fast as it can, and introducing multiple
connections to the same server will only slow things down.

Likewise, if the HTTP server does throttle each connection, but your
Internet connection is so slow that it's not even as fast as the throttled
connection, multiple connections to the same server will again slow things
down.

And of course if the HTTP server is configured to throttle the transfer
for each connection, it is often also configured to disallow multiple
connections from the same client IP address.

There are lots of situations in which a "download manager" won't help at
all. It's one of the reasons I don't bother with them...their ability to
improve things is greatly overstated IMHO.

But it is true that there are scenarios in which multiple connections
retrieving the same file, either from the same HTTP server or from
multiple mirror servers, can indeed improve throughput. These scenarios
may not be very common, but for some people they occur often enough to
make it worthwhile using software that takes advantage of them.

That's true, but these will use dedicated protocols don't they? The
clients also should have multiple NIC's installed connected over
segmented LAN's and/or routers to take some speed advantage of the
parallelism.

It just depends. One well-known example of a dedicated protocol to do
this sort of thing is BitTorrent. And you're right in suggesting that
when this technique is used, it's not always via HTTP. But as you can
tell from the success of BitTorrent, you don't actually need multiple NICs
installed to take advantage of the technique, nor any special hardware at
all.

The goal is to saturate your inbound network connection. If a server is
throttling i/o (or is otherwise restricted) to a level below your inbound
network connection (something that is becoming more and more common as
broadband connections get faster and faster) then having that server send
data on multiple connections (assuming it's not smart enough to detect and
prevent that condition), or having multiple servers each sending different
portions of the same data to the same client, can help saturate that
inbound network connection, minimizing the time it takes to download 100%
of the data requested.

Pete

 

[Jon Skeet [C# MVP]]

A *single file* download from an FTP or HTTP server? How do you
indicate what chunk of the file you want when say using FTP? As far
as I know this is not part of the FTP neither of HTTP protocol.

Content-Range for HTTP
(see http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.16)

RESTART (REST) for FTP I believe (followed by disconnecting after
you've fetched the chunk you want).

Per file? I mean, retrieve a *single file* from multiple servers in
chunks? Say the first 10 MB's from server A the next 2MB from server
B, the next 3 MB from server C? Never heard of something like this.
All I know that there are download managers that download file A from
server A, file B from server B etc.. in case of multi-file downloads.

Nope, there are those which can fetch different bits from different
mirrors. After all, it makes perfect sense to do so if you know they're
the same file.

I used to use one such tool, but I can't remember which off-hand.

 

[Jon Skeet [C# MVP]]

I know that HTTP 1.1 supports range requests, but this is not exactly my point.
The multi part requests in HTTP1.1 are meant to request (for very specic application
purposes) a single part or multiple parts in a single request, but you can't (AFAIK)
requests multiple parts in parallel from multiple client threads.

Why not? They'd be multiple requests, each requesting a single part to
form a whole. What's to stop it working?

That's true, but these will use dedicated protocols don't they? The
clients also should have multiple NIC's installed connected over
segmented LAN's and/or routers to take some speed advantage of the
parallelism.

Why shouldn't it work over multiple HTTP servers (using range headers)?
It depends where the network bottleneck is, of course, but if I can get
from the UK to (say) the US with 1Mbps and to Africa with 1Mbps, in
parallel, it makes sense to fetch at an overall rate of 2Mbps rather
than 1...

 

[Willy Denoyette [MVP]]

Well, I've never actually tried it myself. So I will refrain from claiming that I know
firsthand that this generally works. However, the so-called "download managers" all
claim to work with HTTP servers. I have no reason to doubt them, and based on what I
know about HTTP (which is an extremely simple protocol) it seems likely.

I didn't try it either, so I won't cliam it doesn't work, We did some work at this level
using DECNet/OSI connecting to shared nothing clusters running OpenVMS some years ago, we
were beaten by the HW evolution each step we made, got frustrated and finally gave up to do
this in software,.
I know that down-load managers claim to work over HTTP, but that doesn't mean they support
multi-part parallel request handling over the same or multiple connections, I don't even
know if the protocol allows you to issue new range request when you have a range requests
pending (on the same logical connection).

There is no theoretical reason why it wouldn't work. There's nothing about HTTP that
requires servers to restrict their communications to a given client to a single
connection, and there's nothing about HTTP that stipulates that an HTTP server needs to
coordinate communications on independent connections. If on one connection the client
asks for the first megabyte and on a second connection the same client asks for the
second megabyte, then if the server is capable of servicing both requests at the same
time, there's no reason the client can't wind up receiving both the first and second
megabytes in parallel.

Agreed, but what's the advantage in a simple client server scenario? With simple I mean a
simple PC connected over a dedicated LAN to an HTTP server.

Jon has already outlined the scenarios in which doing so would be helpful. You are
absolutely correct that in many cases, the HTTP server is already providing data as fast
as it can, and introducing multiple connections to the same server will only slow things
down.
Agreed.

Likewise, if the HTTP server does throttle each connection, but your Internet connection
is so slow that it's not even as fast as the throttled connection, multiple connections
to the same server will again slow things down.

Sure.

And of course if the HTTP server is configured to throttle the transfer for each
connection, it is often also configured to disallow multiple connections from the same
client IP address.

They better do ;-)

There are lots of situations in which a "download manager" won't help at all. It's one
of the reasons I don't bother with them...their ability to improve things is greatly
overstated IMHO.

But it is true that there are scenarios in which multiple connections retrieving the same
file, either from the same HTTP server or from multiple mirror servers, can indeed
improve throughput. These scenarios may not be very common, but for some people they
occur often enough to make it worthwhile using software that takes advantage of them.


It just depends. One well-known example of a dedicated protocol to do this sort of thing
is BitTorrent. And you're right in suggesting that when this technique is used, it's not
always via HTTP. But as you can tell from the success of BitTorrent, you don't actually
need multiple NICs installed to take advantage of the technique, nor any special hardware
at all.

The goal is to saturate your inbound network connection. If a server is throttling i/o
(or is otherwise restricted) to a level below your inbound network connection (something
that is becoming more and more common as broadband connections get faster and faster)
then having that server send data on multiple connections (assuming it's not smart enough
to detect and prevent that condition), or having multiple servers each sending different
portions of the same data to the same client, can help saturate that inbound network
connection, minimizing the time it takes to download 100% of the data requested.

Agreed, however, I don't know if this discussion is of any help to the OP, let's see if he
will come back.


Willy.

 

[Jon Skeet [C# MVP]]

I know that down-load managers claim to work over HTTP, but that
doesn't mean they support multi-part parallel request handling over
the same or multiple connections, I don't even know if the protocol
allows you to issue new range request when you have a range requests
pending (on the same logical connection).

I'm not sure what you mean by a "logical" connection, but you can
certainly do it over multiple "real" TCP connections.

Agreed, but what's the advantage in a simple client server scenario?
With simple I mean a simple PC connected over a dedicated LAN to an
HTTP server.

Agreed, it's not useful in that case.

They better do ;-)

Most shouldn't, IMO. If I connect to the BBC and download the home page
which has multiple small images in, it makes sense to use a few
connections to get those images in parallel, given that a lot of the
time will be taken by latency rather than bandwidth.

Disallowing multiple connections from the same IP address to fetch the
same file would make sense, or having a limit on the number of
connections to allow from each IP address.

<snip>

 

[Joerg Jooss]

Thus wrote Willy Denoyette [MVP],

I know that HTTP 1.1 supports range requests, but this is not exactly
my point.
The multi part requests in HTTP1.1 are meant to request (for very
specic application
purposes) a single part or multiple parts in a single request, but
you can't (AFAIK)
requests multiple parts in parallel from multiple client threads.

There's nothing in RFC 2616 that forbids that. The actual limitation is the
maximum number of HTTP connections a client is permitted to establish to
a single host. The spec recommends two.

Cheers,

 

[Willy Denoyette [MVP]]

I'm not sure what you mean by a "logical" connection, but you can
certainly do it over multiple "real" TCP connections.

Sorry, I'm talking about connections in terms of ISO layers, a logical connection is an end
to end socket connection, a connection at the session layer (HTTP is operating at this
layer - and it's called a session). I would call a "real connection", a TCP connection, that
is a connection at the transport layer (connected to a 'port' in TCP/IP terms), say a
connection to www.contoso.com port 80.

Agreed, you can have multiple range requests on separate "real connections" or "logical"
(same or different port, different sockets), however, I'm not sure you can have overlapping
range requests on a *single* an HTTP "session".


a connection with a client connection is the connection

Agreed, it's not useful in that case.


Most shouldn't, IMO. If I connect to the BBC and download the home page
which has multiple small images in, it makes sense to use a few
connections to get those images in parallel, given that a lot of the
time will be taken by latency rather than bandwidth.

Yep, but IMO the different images will be requested by different ovelapped get requests (not
range requests) over a single connection or at most two connection as per HTTP1.1 (see
below) .

Disallowing multiple connections from the same IP address to fetch the
same file would make sense, or having a limit on the number of
connections to allow from each IP address.

This limit is advised by HTTP1.1 (a configurable default for a client obeying the terms of
the protocol) to be a max. of 2 connections to the same server address. We have our backbone
routers configured to impose this maximum, in order to prevent some people to monopolize the
network resources.

Willy.

 

[Willy Denoyette [MVP]]

Thus wrote Willy Denoyette [MVP],


There's nothing in RFC 2616 that forbids that. The actual limitation is the maximum number
of HTTP connections a client is permitted to establish to a single host. The spec
recommends two.

Cheers,

I'm not talking about parallel connections, I'm talking about "same connection" multiple
"overlapping range" requests. I know you kan have parallel requests over two different
connections.

Willy.

 

[Jon Skeet [C# MVP]]

Sorry, I'm talking about connections in terms of ISO layers, a
logical connection is an end to end socket connection, a connection
at the session layer (HTTP is operating at this layer - and it's
called a session). I would call a "real connection", a TCP
connection, that is a connection at the transport layer (connected to
a 'port' in TCP/IP terms), say a connection to www.contoso.com port
80.

Agreed, you can have multiple range requests on separate "real
connections" or "logical" (same or different port, different
sockets), however, I'm not sure you can have overlapping range
requests on a *single* an HTTP "session".

Agreed, but I'm not sure anyone suggested that as an option.

Yep, but IMO the different images will be requested by different
ovelapped get requests (not range requests) over a single connection
or at most two connection as per HTTP1.1 (see below) .

How would you do overlapping get requests on a single connection, out
of interest? That's something I haven't come across (and it sounds
pretty horrendous). Keeping one connection alive for multiple requests,
sure - but I haven't come across overlapping requests on one
connection.

This limit is advised by HTTP1.1 (a configurable default for a client
obeying the terms of the protocol) to be a max. of 2 connections to
the same server address. We have our backbone routers configured to
impose this maximum, in order to prevent some people to monopolize
the network resources.

That's reasonable. I wouldn't like to fix it to 1 though, which is what
it sounded like you were suggesting before.

 

[Jon Skeet [C# MVP]]

I'm not talking about parallel connections, I'm talking about "same
connection" multiple "overlapping range" requests. I know you kan
have parallel requests over two different connections.

But I can't see why you're focusing on "same connection" multiple
requests. There's nothing in the OP's original post to suggest he's
wanting that, and I don't think Peter, Joerg or I have suggested doing
it either.

 

[Willy Denoyette [MVP]]

Why not? They'd be multiple requests, each requesting a single part to
form a whole. What's to stop it working?

I meant - multiple parallel range requests over the same client/server connection here.
Or...
One client A --- one Server,
two client threads,
sharing the same socket connection,
thread 1 issues asynchronous range request x...
thread 2 issues asynchronous range request y...
server accepts request x and at the same time accepts request y, starts to return data
for x and y.
AFAIK above is not possible, that doesn't mean it *fails* visibly.
IMO the server will choose to handle request x and queue request y, that means he will
return the data for x before he starts to return data for y.
Again, I'm not pretending this is exactly the case, I'll see if I can build a test case.

Why shouldn't it work over multiple HTTP servers (using range headers)?
It depends where the network bottleneck is, of course, but if I can get
from the UK to (say) the US with 1Mbps and to Africa with 1Mbps, in
parallel, it makes sense to fetch at an overall rate of 2Mbps rather
than 1...

Yep, but say you have 1Mbps to the US and 50Kbps to Afrika, is your download manager clever
enough to request all chunks from the US instead of waiting for a chunk to arrive from
Afrika, or is he clever enough to cancel the request and re-issue the same request over the
US connection. This requires permanent real-time monitoring of the transfer rates, something
that can't reliably be done at the application layer, that's what I meant when I said you
really need sophisticated software and be sure to measure (as always) the real benefits, I
guess (expensive) download managers can do it while other pretend they can, big difference!.

Willy.

 

[Jon Skeet [C# MVP]]

I meant - multiple parallel range requests over the same client/server connection here.
Or...
One client A --- one Server,
two client threads,
sharing the same socket connection,

And that's the problem - I don't *think* anyone other than you has a
single connection in mind.

thread 1 issues asynchronous range request x...
thread 2 issues asynchronous range request y...
server accepts request x and at the same time accepts request y,
starts to return data for x and y.

AFAIK above is not possible, that doesn't mean it *fails* visibly.
IMO the server will choose to handle request x and queue request y,
that means he will return the data for x before he starts to return
data for y. Again, I'm not pretending this is exactly the case, I'll
see if I can build a test case.

It seems likely - the server probably wouldn't bother reading until
ithad finished writing. Of course, if you're really unlucky the two
requests will be sent at the same time and corrupt each other

Yep, but say you have 1Mbps to the US and 50Kbps to Afrika, is your
download manager clever enough to request all chunks from the US
instead of waiting for a chunk to arrive from Afrika, or is he clever
enough to cancel the request and re-issue the same request over the
US connection. This requires permanent real-time monitoring of the
transfer rates, something that can't reliably be done at the
application layer, that's what I meant when I said you really need
sophisticated software and be sure to measure (as always) the real
benefits, I guess (expensive) download managers can do it while other
pretend they can, big difference!.

Well, you'd have to have a certain amount of control of the client
implementation - e.g. not waiting until a whole response was ready
before returning - but I don't think it's that hard to do. I think most
of these download managers try a short sample request once to start
with to work out which servers to bother with at all, and how much to
ask each one to do. I don't know whether they'd go as far as cancelling
a chunk they'd only received partially, but it's possible.

I don't bother with such things these days, but a while ago (on a dial-
up line, even, where you'd think I'd always be able to saturate the
last mile) it was genuinely useful.

 

[Willy Denoyette [MVP]]

But I can't see why you're focusing on "same connection" multiple
requests. There's nothing in the OP's original post to suggest he's
wanting that, and I don't think Peter, Joerg or I have suggested doing
it either.

Wait a minute I guess your purpose is one again to pick on me, right?
The OP didn't even mention HTTP or FTP nor did I, it was you you started with this FTP and
HTTP protocol stuff, and it was Peter who started (rightfully) with the HTTP range requests.
All I said, in a reply to Peter, that AFAIK you can't issue multiple range requests in
parallel over the same connection, so why would I even suggest this? I don't even know
whether the OP isn't simply talking about a file server request using System.IO.

Willy.