file transfer with sockets

[David]

I have googled to no avail on getting specifically what I'm looking for. I
have found plenty of full blown apps that implement some type of file
transfer but what I'm specifcally looking for is an example to follow for
using a tcp socket to transfer files between client/server, server/client.
Both server and client are my program so I'm not looking for how to
implement an FTP client, or how to download a file from a web server via
http etc... Protocol between my client/server is a simple command based
structure and I intended on it going something like this in english:

this example is client to server scenario but I'll implement on both ends..
so client/server can be server/client etc..
1) client requests a file from server
2) server verifies file exists and sends back an 'ok' with the files size
(or message saying no file exists)
3) client receives file's size and sends the 'ok' to start the transfer
4) server sends file / client receives file
5) client sends 'ok' after receive is done

I've given these steps here to be clear and specific on the part I'm trying
to learn how to do. I can do all steps here fine except step 4, the actual
send/receive of the file. Help on that would be greatly appreciated.

I'm not even sure if those specific steps will stay like that, I only used
them to narrow down the area of my question. Further, I'm looking to do this
with the sockets directly, using socket.send and socket.receive, (or their
async methods), as I've read elseware that is the most effecient way, as
opposed to using a networkstream object. Aside from help on the specific
task I'm trying to learn, any comments on which technique is a good choice
are also welcome (socket.send etc... networkstream object.. and I honestly
just saw a 'socket.sendfile' method in VS help before finishing drafting
this post... I'm posting anyway since I learn a lot from this forum, and
even if one of the higher level classes is a better choice, I want to learn
the socket way first.)

 

[Nicholas Paldino [.NET/C# MVP]]

David,

You don't have to use the NetworkStream object, you can use a Socket
object just fine. Once you have the file on the server side, you can just
read the bytes from the file in chunks, and then pass them to the Send
method, sending the bytes over the stream. Of course, your client has to
read them, and it has to read only the number of bytes that you are going to
send (which it already knows because you sent the length already).

 

[Ignacio Machin \( .NET/ C# MVP \)]

Hi,

this example is client to server scenario but I'll implement on both
ends.. so client/server can be server/client etc..
1) client requests a file from server
2) server verifies file exists and sends back an 'ok' with the files size
(or message saying no file exists)
3) client receives file's size and sends the 'ok' to start the transfer
4) server sends file / client receives file
5) client sends 'ok' after receive is done

Sounds like a workable protocol to me

I'm not even sure if those specific steps will stay like that, I only used
them to narrow down the area of my question. Further, I'm looking to do
this with the sockets directly, using socket.send and socket.receive, (or
their async methods)

Start with the sync.ed method first, IMO async should only be used in
interactive applications. In the same way you can have a sync server and an
async client.

, as I've read elseware that is the most effecient way, as opposed to using
a networkstream object. Aside from help on the specific task I'm trying to
learn, any comments on which technique is a good choice are also welcome
(socket.send etc... networkstream object..

Just use Socket Send/Receive, it should be trivial to implement. just a
simple loop whiel you have bytes to read (you already knowthe file size).

 

[David]

Hi Nicholas,
"Once you have the file on the server side, you can just
read the bytes from the file in chunks, and then pass them to the Send
method, sending the bytes over the stream. Of course, your client has to
read them, and it has to read only the number of bytes that you are going to
send (which it already knows because you sent the length already)."

I don't know what that would actually look like on sending and receiving
end? However I get the concept.
I'm embarrassed to say but I really haven't used .net's streaming IO model
yet... I used vb4 a long time ago, my programming activities now are mostly
scripting for automation related to network administration. I'm just now
really trying to get into c#/.net/network programming. I get that all .net
IO uses streams, and that I would use a stream (or fileStream?) to read the
file from disk on server, assuming reading it into a byte buffer, then using
the socket to send that byte buffer. But how do I, for example, control and
keep track of the 'chunks', you mentioned? would it be one call to
socket.send after accumulating all the file chunks or a loop resulting in
multiple calls to socket.send? I assume a loop is more likely so it would
work on small and large files... unfortunately I am still at the beginning
learning stages of this and lack some crucial fundamentals. But I can pick
it up real quick because of past programming experience and some admin
scripting.

David,

You don't have to use the NetworkStream object, you can use a Socket
object just fine. Once you have the file on the server side, you can just
read the bytes from the file in chunks, and then pass them to the Send
method, sending the bytes over the stream. Of course, your client has to
read them, and it has to read only the number of bytes that you are going
to send (which it already knows because you sent the length already).


--
- Nicholas Paldino [.NET/C# MVP]
- (e-mail address removed)

I have googled to no avail on getting specifically what I'm looking for. I
have found plenty of full blown apps that implement some type of file
transfer but what I'm specifcally looking for is an example to follow for
using a tcp socket to transfer files between client/server, server/client.
Both server and client are my program so I'm not looking for how to
implement an FTP client, or how to download a file from a web server via
http etc... Protocol between my client/server is a simple command based
structure and I intended on it going something like this in english:

this example is client to server scenario but I'll implement on both
ends.. so client/server can be server/client etc..
1) client requests a file from server
2) server verifies file exists and sends back an 'ok' with the files size
(or message saying no file exists)
3) client receives file's size and sends the 'ok' to start the transfer
4) server sends file / client receives file
5) client sends 'ok' after receive is done

I've given these steps here to be clear and specific on the part I'm
trying to learn how to do. I can do all steps here fine except step 4,
the actual send/receive of the file. Help on that would be greatly
appreciated.

I'm not even sure if those specific steps will stay like that, I only
used them to narrow down the area of my question. Further, I'm looking to
do this with the sockets directly, using socket.send and socket.receive,
(or their async methods), as I've read elseware that is the most
effecient way, as opposed to using a networkstream object. Aside from
help on the specific task I'm trying to learn, any comments on which
technique is a good choice are also welcome (socket.send etc...
networkstream object.. and I honestly just saw a 'socket.sendfile' method
in VS help before finishing drafting this post... I'm posting anyway
since I learn a lot from this forum, and even if one of the higher level
classes is a better choice, I want to learn the socket way first.)

 

[Nicholas Paldino [.NET/C# MVP]]

David,

Yes, you would use a filestream. You would make calls to Read on the
filestream instance, reading say, 1KB of bytes (or any other number you feel
is appropriate depending on the size of the files you will typically send)
at a time, and then passing the byte array that you read the contents into
to the Send method on the socket.

You could load the entire contents of the file into one big byte array,
but that wouldn't be too efficient, and would gobble up memory for large
files (hence the reading in chunks).


--
- Nicholas Paldino [.NET/C# MVP]
- (e-mail address removed)

Hi Nicholas,
"Once you have the file on the server side, you can just
read the bytes from the file in chunks, and then pass them to the Send
method, sending the bytes over the stream. Of course, your client has to
read them, and it has to read only the number of bytes that you are going
to
send (which it already knows because you sent the length already)."

I don't know what that would actually look like on sending and receiving
end? However I get the concept.
I'm embarrassed to say but I really haven't used .net's streaming IO model
yet... I used vb4 a long time ago, my programming activities now are
mostly scripting for automation related to network administration. I'm
just now really trying to get into c#/.net/network programming. I get that
all .net IO uses streams, and that I would use a stream (or fileStream?)
to read the file from disk on server, assuming reading it into a byte
buffer, then using the socket to send that byte buffer. But how do I, for
example, control and keep track of the 'chunks', you mentioned? would it
be one call to socket.send after accumulating all the file chunks or a
loop resulting in multiple calls to socket.send? I assume a loop is more
likely so it would work on small and large files... unfortunately I am
still at the beginning learning stages of this and lack some crucial
fundamentals. But I can pick it up real quick because of past programming
experience and some admin scripting.

David,

You don't have to use the NetworkStream object, you can use a Socket
object just fine. Once you have the file on the server side, you can
just read the bytes from the file in chunks, and then pass them to the
Send method, sending the bytes over the stream. Of course, your client
has to read them, and it has to read only the number of bytes that you
are going to send (which it already knows because you sent the length
already).


--
- Nicholas Paldino [.NET/C# MVP]
- (e-mail address removed)

I have googled to no avail on getting specifically what I'm looking for.
I have found plenty of full blown apps that implement some type of file
transfer but what I'm specifcally looking for is an example to follow for
using a tcp socket to transfer files between client/server,
server/client. Both server and client are my program so I'm not looking
for how to implement an FTP client, or how to download a file from a web
server via http etc... Protocol between my client/server is a simple
command based structure and I intended on it going something like this in
english:

this example is client to server scenario but I'll implement on both
ends.. so client/server can be server/client etc..
1) client requests a file from server
2) server verifies file exists and sends back an 'ok' with the files
size (or message saying no file exists)
3) client receives file's size and sends the 'ok' to start the transfer
4) server sends file / client receives file
5) client sends 'ok' after receive is done

I've given these steps here to be clear and specific on the part I'm
trying to learn how to do. I can do all steps here fine except step 4,
the actual send/receive of the file. Help on that would be greatly
appreciated.

I'm not even sure if those specific steps will stay like that, I only
used them to narrow down the area of my question. Further, I'm looking
to do this with the sockets directly, using socket.send and
socket.receive, (or their async methods), as I've read elseware that is
the most effecient way, as opposed to using a networkstream object.
Aside from help on the specific task I'm trying to learn, any comments
on which technique is a good choice are also welcome (socket.send etc...
networkstream object.. and I honestly just saw a 'socket.sendfile'
method in VS help before finishing drafting this post... I'm posting
anyway since I learn a lot from this forum, and even if one of the
higher level classes is a better choice, I want to learn the socket way
first.)

 

[Peter Duniho]

[...]
You could load the entire contents of the file into one big byte
array, but that wouldn't be too efficient, and would gobble up memory
for large files (hence the reading in chunks).

And of course would not even work for files too large to fit in the
process's address space.

 

[David]

I've been reading up on streams and file I/O to make this come together. Two
questions:

1) should I just send each file chunk consecutively or wait for a
confirmation from client for each chunk before sending the next?

2) my current structure is a simple command/response setup. I send command
to server, server interprets command, executes it, then simply returns
results of the command... I did this using asynch calls for both client and
server. Is it ok to use the synch calls now from within my asynch callbacks
for the file transfer? a read a book that said something about not mixing
the asynch and synch calls? starving the threadpool? I'm in over my head but
making progress

David,

Yes, you would use a filestream. You would make calls to Read on the
filestream instance, reading say, 1KB of bytes (or any other number you
feel is appropriate depending on the size of the files you will typically
send) at a time, and then passing the byte array that you read the
contents into to the Send method on the socket.

You could load the entire contents of the file into one big byte array,
but that wouldn't be too efficient, and would gobble up memory for large
files (hence the reading in chunks).


--
- Nicholas Paldino [.NET/C# MVP]
- (e-mail address removed)

Hi Nicholas,
"Once you have the file on the server side, you can just
read the bytes from the file in chunks, and then pass them to the Send
method, sending the bytes over the stream. Of course, your client has to
read them, and it has to read only the number of bytes that you are going
to
send (which it already knows because you sent the length already)."

I don't know what that would actually look like on sending and receiving
end? However I get the concept.
I'm embarrassed to say but I really haven't used .net's streaming IO
model yet... I used vb4 a long time ago, my programming activities now
are mostly scripting for automation related to network administration.
I'm just now really trying to get into c#/.net/network programming. I get
that all .net IO uses streams, and that I would use a stream (or
fileStream?) to read the file from disk on server, assuming reading it
into a byte buffer, then using the socket to send that byte buffer. But
how do I, for example, control and keep track of the 'chunks', you
mentioned? would it be one call to socket.send after accumulating all the
file chunks or a loop resulting in multiple calls to socket.send? I
assume a loop is more likely so it would work on small and large files...
unfortunately I am still at the beginning learning stages of this and
lack some crucial fundamentals. But I can pick it up real quick because
of past programming experience and some admin scripting.

David,

You don't have to use the NetworkStream object, you can use a Socket
object just fine. Once you have the file on the server side, you can
just read the bytes from the file in chunks, and then pass them to the
Send method, sending the bytes over the stream. Of course, your client
has to read them, and it has to read only the number of bytes that you
are going to send (which it already knows because you sent the length
already).


--
- Nicholas Paldino [.NET/C# MVP]
- (e-mail address removed)

I have googled to no avail on getting specifically what I'm looking for.
I have found plenty of full blown apps that implement some type of file
transfer but what I'm specifcally looking for is an example to follow
for using a tcp socket to transfer files between client/server,
server/client. Both server and client are my program so I'm not looking
for how to implement an FTP client, or how to download a file from a web
server via http etc... Protocol between my client/server is a simple
command based structure and I intended on it going something like this
in english:

this example is client to server scenario but I'll implement on both
ends.. so client/server can be server/client etc..
1) client requests a file from server
2) server verifies file exists and sends back an 'ok' with the files
size (or message saying no file exists)
3) client receives file's size and sends the 'ok' to start the transfer
4) server sends file / client receives file
5) client sends 'ok' after receive is done

I've given these steps here to be clear and specific on the part I'm
trying to learn how to do. I can do all steps here fine except step 4,
the actual send/receive of the file. Help on that would be greatly
appreciated.

I'm not even sure if those specific steps will stay like that, I only
used them to narrow down the area of my question. Further, I'm looking
to do this with the sockets directly, using socket.send and
socket.receive, (or their async methods), as I've read elseware that is
the most effecient way, as opposed to using a networkstream object.
Aside from help on the specific task I'm trying to learn, any comments
on which technique is a good choice are also welcome (socket.send
etc... networkstream object.. and I honestly just saw a
'socket.sendfile' method in VS help before finishing drafting this
post... I'm posting anyway since I learn a lot from this forum, and
even if one of the higher level classes is a better choice, I want to
learn the socket way first.)

 

[Peter Duniho]

1) should I just send each file chunk consecutively or wait for a
confirmation from client for each chunk before sending the next?

You should just send it. In fact, you should not really need to get
confirmation for any chunk of data at all. Armed with the knowledge of
how long the file being sent is, the receiver can just keep receiving data
until they've received that many bytes, and then send a single
confirmation at the very end (if desired...many file transfer
implementations don't even bother to do this, since the client can
determine for itself whether all the bytes were sent and do something
appropriate if they weren't).

2) my current structure is a simple command/response setup. I send
command
to server, server interprets command, executes it, then simply returns
results of the command... I did this using asynch calls for both client
and server. Is it ok to use the synch calls now from within my asynch
callbacks for the file transfer? a read a book that said something about
not mixing the asynch and synch calls? starving the threadpool? I'm in
over my head but making progress

I'm not really sure what the book you read is discussing, but I can see
how if in the callback for an asynch method, you then called some synch
method, that could cause the callback to block and thus tie up a
threadpool thread. With enough threadpool threads blocked waiting on
synchronous methods, that could indeed cause you to run out of threadpool
threads (of course, for this to happen you'd need more than one Socket
with which to create outstanding asynchronous i/o requests, and it doesn't
sound like that would happen in your case...still, it's not good design).

So, no...I don't think you should use synchronous Socket calls from within
your callback. Now, that doesn't mean you can't call *any* synchronous
calls. Even a synchronous call to read more data from your file stream so
that you can send another chunk of data should be fine. Yes, the file i/o
would be done synchronously, but relative to the network the disk i/o
should be extremely fast and shouldn't be a problem.

Pete

 

[David]

thanks Peter, sorry for the delayed response.. I was out of the office a few
days. I appreciate the help. A question has come to mind that I didn't think
about until exploring transferring files because up to this point I have
only sent small amounts of data back and forth... the asynch model requires
callbacks... below is one of the callbacks I'm using for receiving data. Its
based on an example I got from MSDN. Notice how within the callback method a
new socket.beginReceive is initiated. The first socket.beginReceive is
called from a different method, which obviously calls this callback method,
then within this callback method, another socket.beginReceive is called..
and so on until all data is received. Kind of calling itself recursively.

My questions: when this happens, is a new threadpool thread taken each time
the callback is called within this iterative process? I am thinking about
this because I'm figuring out how I should handle file transfers, but also,
just trying to further my understanding in general, of socket communication
and the asynch model. I'm wondering about, in the case of large amounts of
data (like a file transfer), the possible overhead of taking a new thread
each time.. possibly hundreds of iterations transferring a large file? Below
is example of receiving end but my same questions apply to sending end as
well... is it ok to loop through these asynch calls for transferring large
amounts of data? or should I be setting up one new thread myself and using
the synchronous socket send/receive calls just for the large transfers?

private static void ReceiveCallback(IAsyncResult ar)
{
try
{
// Retrieve my state object
StateObject state = (StateObject)ar.AsyncState;

int bytesRead = state.clientSocket.EndReceive(ar);

if (bytesRead > 0)
{
// There might be more data, so store the data received
so far.
state.sb.Append(Encoding.ASCII.GetString(state.buffer,
0, bytesRead));

// Get the rest of the data.
state.clientSocket.BeginReceive(state.buffer, 0,
state.bufferSize, 0, new AsyncCallback(ReceiveCallback), state);
}
else
{
// All the data has arrived;
// Signal that all bytes have been received.
g_receiveDone.Set();
}
}
catch (Exception e)
{
Console.WriteLine(e.ToString());
}
}

 

[Peter Duniho]

[...] Notice how within the callback method a
new socket.beginReceive is initiated. The first socket.beginReceive is
called from a different method, which obviously calls this callback
method,
then within this callback method, another socket.beginReceive is called..
and so on until all data is received. Kind of calling itself recursively.

My questions: when this happens, is a new threadpool thread taken each
time the callback is called within this iterative process?

That's fine. It is true that when you post a new buffer for receiving (or
sending), it's possible that receive could complete causing another thread
to run, but a) as long as you don't take too much time in the current
callback, it's likely your thread will finish before, or at least have at
most one more timeslice left when, the other thread runs and b) since i/o
is so much slower than what the CPU is capable of dealing with, it's very
unlikely that you could keep enough threads busy handling i/o to actually
run out of thread-pool threads. I could see having a few threads all
active at once, but at some point you're going to run out of data, all
those threads will complete, and the pool will be idle until the next bit
of data shows up.

It's important to understand that calling BeginReceive does not assign a
thread for execution right away. It posts a buffer assigned to an i/o
completion port, and not until the i/o on that port has completed will any
thread actually be assigned to handle the completion event. In other
words, no i/o, no thread running, and if there is i/o then you DO want a
thread to run and handle the i/o as soon as you can.

The paradigm you're seeing in that sample code is the .NET variation of
using Winsock with i/o completion ports, which is the most efficient, most
scalable version of network i/o that Windows has to offer. You should be
able to confidently use the async pattern for the .NET Socket class,
following the example code, without worrying about using up your thread
pool.

Pete

 

[David]

ok, thanks Peter!

[...] Notice how within the callback method a
new socket.beginReceive is initiated. The first socket.beginReceive is
called from a different method, which obviously calls this callback
method,
then within this callback method, another socket.beginReceive is called..
and so on until all data is received. Kind of calling itself recursively.

My questions: when this happens, is a new threadpool thread taken each
time the callback is called within this iterative process?

That's fine. It is true that when you post a new buffer for receiving (or
sending), it's possible that receive could complete causing another thread
to run, but a) as long as you don't take too much time in the current
callback, it's likely your thread will finish before, or at least have at
most one more timeslice left when, the other thread runs and b) since i/o
is so much slower than what the CPU is capable of dealing with, it's very
unlikely that you could keep enough threads busy handling i/o to actually
run out of thread-pool threads. I could see having a few threads all
active at once, but at some point you're going to run out of data, all
those threads will complete, and the pool will be idle until the next bit
of data shows up.

It's important to understand that calling BeginReceive does not assign a
thread for execution right away. It posts a buffer assigned to an i/o
completion port, and not until the i/o on that port has completed will any
thread actually be assigned to handle the completion event. In other
words, no i/o, no thread running, and if there is i/o then you DO want a
thread to run and handle the i/o as soon as you can.

The paradigm you're seeing in that sample code is the .NET variation of
using Winsock with i/o completion ports, which is the most efficient, most
scalable version of network i/o that Windows has to offer. You should be
able to confidently use the async pattern for the .NET Socket class,
following the example code, without worrying about using up your thread
pool.

Pete