Hi,

I posted a question a while back relating to sending data over a
network, and was told to go and look into how networks send data.

I know that TCP can not guarentee the order in which data is delivered
to a device. In light of this, I have a problem which I can not find a
way out of:

I need to send a file of around 4-5Mb over a network. To prevent
communicaiton thread being locked, I was attempting to segment the
data into 8k segments of data to be sent. In the header of each
segment, I was attaching a 4byte integer representing the segment
number, which will allow the reciever to place the data in the
relevant position in the output file. Occasionally the client decodes
the segment identifier into a bizzare value (last time I tried this, I
was expecting to see segment number 234, and recieved -432352 or
similar).

However, this is not working,and the data being written out to a file
at the reciever is not the same as the input data at the server. Why
would this be?

Please can anyone shed any light on this matter, or show me an example
of how I should send the data/ensure that the data at the client end
is pieced together correctly?

Many thanks for any help - I am very confused by this matter.

David

Well,

No, TCP *does* guarantee delivery order. UDP does not. If you establish a
socket connection between two points, then the order in which data is sent
from one end *will be* the order in which it is received at the other end.

In your particular case, just send a header saying "Here comes a file that's
xxxx bytes long", then send the entire content of the file (being sure to
send the right number of bytes). On the other end, receive the header,
then, using the indicated number of bytes, start receiving and writing to
the file. Breaking up the file into pieces and sending them in separate
sends *is* a good idea, though. You don't have to put a header with every
piece, though.

Paul T.

"David Glover" <(E-Mail Removed)> wrote in message
news:(E-Mail Removed)...
> Hi,
>
> I posted a question a while back relating to sending data over a
> network, and was told to go and look into how networks send data.
>
> I know that TCP can not guarentee the order in which data is delivered
> to a device. In light of this, I have a problem which I can not find a
> way out of:
>
> I need to send a file of around 4-5Mb over a network. To prevent
> communicaiton thread being locked, I was attempting to segment the
> data into 8k segments of data to be sent. In the header of each
> segment, I was attaching a 4byte integer representing the segment
> number, which will allow the reciever to place the data in the
> relevant position in the output file. Occasionally the client decodes
> the segment identifier into a bizzare value (last time I tried this, I
> was expecting to see segment number 234, and recieved -432352 or
> similar).
>
> However, this is not working,and the data being written out to a file
> at the reciever is not the same as the input data at the server. Why
> would this be?
>
> Please can anyone shed any light on this matter, or show me an example
> of how I should send the data/ensure that the data at the client end
> is pieced together correctly?
>
> Many thanks for any help - I am very confused by this matter.
>
> David

Hi,

Paul's answer is correct. TCP "assures" (assurance is just high
probability, but it should be good enough) that packets will be delivered,
and that delivery will be in the order of transmission. There may be
reasons to packetize data, and to provide extra error-detection and recovery
mechanisms (such as might be desired if using a wireless connection, for
example). However, in general, it is sufficient to do as Paul suggests --
though I sometimes include a CRC calculation in addition to the file size.

Dick

--
Richard Grier (Microsoft Visual Basic MVP)

See www.hardandsoftware.net for contact information.

Author of Visual Basic Programmer's Guide to Serial Communications, 3rd
Edition ISBN 1-890422-27-4 (391 pages) published February 2002.

Basically the file needs to be segmented to allow a situation where by
the client (which is on a wireless link) uploads a file, and at the same
time, the user presses a button such as "get lastest server
information", which needs to send a message to the server, and then
recieve a message back. Other wise the user will have to wait for the
compelete transmission of the file in question before they can initiate
another operation.

Considering the applicaiton being developed is a distributed network of
files (using MP3 files in this case), the server get a request from a
client for a file off another client. The 2nd client then starts to
upload the file. During this up load the user of the second client
wishes to see what tracks are availible on the system. The user would be
blocked by an operation they did not initiate, and this would be poor
from a usability perspective.

Many thnaks for your input however, I am reluctant to give up this idea
of segmenting the file considering the operating envrionment. This is
being a harder nut to crack that 1st thought.



*** Sent via Developersdex http://www.developersdex.com ***
Don't just participate in USENET...get rewarded for it!

Many thanks for the reply. I fear my initial response to your post may
have been lost. In case it has - here is another response:

I have gone back to basics and written a basic full .NET applicaiton
which will segment the file and send it to a client. The output file and
the input files are different when compared in a hex editor (I am using
HexCmp).

The test app below does not implement segment ID fields, however the
final app will. Can anyone see/suggest what may be going wrong here:

***SENDER APPLICAITON***
...
private Socket clientSocket
...

private void button1_Click(object sender, System.EventArgs e)
{
Socket serverSocket = new Socket(AddressFamily.InterNetwork,
SocketType.Stream,
ProtocolType.Tcp) ;

IPEndPoint localEndPoint = new IPEndPoint(IPAddress.Any, 676) ;
serverSocket.Bind(localEndPoint) ;
serverSocket.Listen(10) ;
clientSocket = serverSocket.Accept() ;
}

private void button2_Click(object sender, System.EventArgs e)
{
bool f = clientSocket.Blocking ;

//NetworkStream ns = new NetworkStream(clientSocket,
FileAccess.Write, true) ;


FileStream fs = new FileStream(@"e:\dump\out.mp3",FileMode.Open,
FileAccess.Read, FileShare.Read) ;
BinaryReader b = new BinaryReader(fs, System.Text.Encoding.ASCII) ;
long dataLength = fs.Length ;

//calculate the number of segments that are required to send the file
int fileSizei = (int)dataLength ;
double fileSized = fileSizei ;
double segmentsd = fileSized / 8192 ;

int segments = Convert.ToInt32(segmentsd) ;
MessageBox.Show("File size : " + fileSizei.ToString()) ;
MessageBox.Show("Segments : " + segments.ToString()) ;


int totalBytesWritten = 0;
for(int i = 0 ; i < segments ; i++)
{
//read the data from the buffer
if(i == (segments-1))
{
//last segment
//calculate the number of bytes left
int bytesLeft = fileSizei - totalBytesWritten ;

byte[] send = b.ReadBytes(bytesLeft) ;

fs.Read(send, 0, bytesLeft) ;

int bytesSent = clientSocket.Send(send, 0, bytesLeft,
SocketFlags.None) ;
Console.WriteLine(bytesSent) ;
if(bytesSent != bytesLeft)
{
MessageBox.Show("ERROR WRITING BYTES") ;
}
}
else
{
//byte[] send = new byte[8192] ;
//fs.Read(send, 0,8192) ;
byte[] send = b.ReadBytes(8192) ;
int sentBytes = clientSocket.Send(send,0,8192,SocketFlags.None) ;
Console.WriteLine(sentBytes) ;
if(sentBytes != 8192)
{
MessageBox.Show("ERROR WRITING 8192 bytes") ;
}
totalBytesWritten += 8192 ;
}
//wait for the "OK"
byte[]ok = new byte[16] ;
clientSocket.Receive(ok, 0, 16, SocketFlags.None) ;


}
//fs.Close() ;
b.Close() ;
MessageBox.Show("SEND COMPLETE") ;
}

***RECIEVER***
...

private Socket sock ;

...
private void button1_Click(object sender, System.EventArgs e)
{
sock = new Socket(AddressFamily.InterNetwork, SocketType.Stream,
ProtocolType.Tcp) ;

IPAddress sip = IPAddress.Parse("192.168.10.2") ;
IPEndPoint remote = new IPEndPoint(sip,676) ;
sock.Connect(remote) ;

}

private void button2_Click(object sender, System.EventArgs e)
{
//make the user enter hte file size and number of segments
//these aer messagebox'd to the screen in the sender
//adn this is only a test harness when all said and done
int fileSize = Int32.Parse(textBox1.Text) ;
int segments = Int32.Parse(textBox2.Text) ;
bool block = sock.Blocking ;
FileStream f = new FileStream(@"c:\recieved.mp3",
FileMode.OpenOrCreate, FileAccess.ReadWrite) ;
BinaryWriter b = new BinaryWriter(f, System.Text.Encoding.ASCII) ;

int totalrecieved = 0;
for(int i = 0 ; i < segments ; i++)
{
if(i == (segments -1))
{
//last segment
int remaining = fileSize - totalrecieved ;
byte[] read = new byte[remaining] ;
sock.Receive(read, 0, remaining, SocketFlags.None) ;
b.Write(read) ;
}
else
{
byte[] read = new byte[8192] ;
sock.Receive(read, 0, 8192, SocketFlags.None) ;

b.Write(read) ;
totalrecieved += 8192;
}

string OK = "OK" ;
byte[] okb = System.Text.Encoding.ASCII.GetBytes(OK) ;
Console.WriteLine(okb.Length) ;
sock.Send(okb, 0, okb.Length, SocketFlags.None) ;
}

b.Close() ;
MessageBox.Show("DONE") ;
//f.Close() ;

}


I have put the 2 files used at:

http://www.lancs.ac.uk/ug/gloverd/out.mp3 - the file used by the SENDER
http://www.lancs.ac.uk/ug/gloverd/recieved.mp3 - the output file from
the reciever. Using a hex comparer, it can be seen that the errors start
on byte 19519.


Many thanks for any help anyone can provide, as I have spent quite a
while getting nowhere on this problem.

Regards and thanks,

David



*** Sent via Developersdex http://www.developersdex.com ***
Don't just participate in USENET...get rewarded for it!

At a quick guess, the encoding might be responsible. It might do things
like convert CR/LF pairs to CR or something along those lines. Maybe the
results of the comparison would tell you that. There might also be problems
with null characters, depending on what the content of the file looks like.

Paul T.

"David Glover" <(E-Mail Removed)> wrote in message
news:(E-Mail Removed)...
> Many thanks for the reply. I fear my initial response to your post may
> have been lost. In case it has - here is another response:
>
> I have gone back to basics and written a basic full .NET applicaiton
> which will segment the file and send it to a client. The output file and
> the input files are different when compared in a hex editor (I am using
> HexCmp).
>
> The test app below does not implement segment ID fields, however the
> final app will. Can anyone see/suggest what may be going wrong here:
>
> ***SENDER APPLICAITON***
> ..
> private Socket clientSocket
> ..
>
> private void button1_Click(object sender, System.EventArgs e)
> {
> Socket serverSocket = new Socket(AddressFamily.InterNetwork,
> SocketType.Stream,
> ProtocolType.Tcp) ;
>
> IPEndPoint localEndPoint = new IPEndPoint(IPAddress.Any, 676) ;
> serverSocket.Bind(localEndPoint) ;
> serverSocket.Listen(10) ;
> clientSocket = serverSocket.Accept() ;
> }
>
> private void button2_Click(object sender, System.EventArgs e)
> {
> bool f = clientSocket.Blocking ;
>
> //NetworkStream ns = new NetworkStream(clientSocket,
> FileAccess.Write, true) ;
>
>
> FileStream fs = new FileStream(@"e:\dump\out.mp3",FileMode.Open,
> FileAccess.Read, FileShare.Read) ;
> BinaryReader b = new BinaryReader(fs, System.Text.Encoding.ASCII) ;
> long dataLength = fs.Length ;
>
> //calculate the number of segments that are required to send the file
> int fileSizei = (int)dataLength ;
> double fileSized = fileSizei ;
> double segmentsd = fileSized / 8192 ;
>
> int segments = Convert.ToInt32(segmentsd) ;
> MessageBox.Show("File size : " + fileSizei.ToString()) ;
> MessageBox.Show("Segments : " + segments.ToString()) ;
>
>
> int totalBytesWritten = 0;
> for(int i = 0 ; i < segments ; i++)
> {
> //read the data from the buffer
> if(i == (segments-1))
> {
> //last segment
> //calculate the number of bytes left
> int bytesLeft = fileSizei - totalBytesWritten ;
>
> byte[] send = b.ReadBytes(bytesLeft) ;
>
> fs.Read(send, 0, bytesLeft) ;
>
> int bytesSent = clientSocket.Send(send, 0, bytesLeft,
> SocketFlags.None) ;
> Console.WriteLine(bytesSent) ;
> if(bytesSent != bytesLeft)
> {
> MessageBox.Show("ERROR WRITING BYTES") ;
> }
> }
> else
> {
> //byte[] send = new byte[8192] ;
> //fs.Read(send, 0,8192) ;
> byte[] send = b.ReadBytes(8192) ;
> int sentBytes = clientSocket.Send(send,0,8192,SocketFlags.None) ;
> Console.WriteLine(sentBytes) ;
> if(sentBytes != 8192)
> {
> MessageBox.Show("ERROR WRITING 8192 bytes") ;
> }
> totalBytesWritten += 8192 ;
> }
> //wait for the "OK"
> byte[]ok = new byte[16] ;
> clientSocket.Receive(ok, 0, 16, SocketFlags.None) ;
>
>
> }
> //fs.Close() ;
> b.Close() ;
> MessageBox.Show("SEND COMPLETE") ;
> }
>
> ***RECIEVER***
> ..
>
> private Socket sock ;
>
> ..
> private void button1_Click(object sender, System.EventArgs e)
> {
> sock = new Socket(AddressFamily.InterNetwork, SocketType.Stream,
> ProtocolType.Tcp) ;
>
> IPAddress sip = IPAddress.Parse("192.168.10.2") ;
> IPEndPoint remote = new IPEndPoint(sip,676) ;
> sock.Connect(remote) ;
>
> }
>
> private void button2_Click(object sender, System.EventArgs e)
> {
> //make the user enter hte file size and number of segments
> //these aer messagebox'd to the screen in the sender
> //adn this is only a test harness when all said and done
> int fileSize = Int32.Parse(textBox1.Text) ;
> int segments = Int32.Parse(textBox2.Text) ;
> bool block = sock.Blocking ;
> FileStream f = new FileStream(@"c:\recieved.mp3",
> FileMode.OpenOrCreate, FileAccess.ReadWrite) ;
> BinaryWriter b = new BinaryWriter(f, System.Text.Encoding.ASCII) ;
>
> int totalrecieved = 0;
> for(int i = 0 ; i < segments ; i++)
> {
> if(i == (segments -1))
> {
> //last segment
> int remaining = fileSize - totalrecieved ;
> byte[] read = new byte[remaining] ;
> sock.Receive(read, 0, remaining, SocketFlags.None) ;
> b.Write(read) ;
> }
> else
> {
> byte[] read = new byte[8192] ;
> sock.Receive(read, 0, 8192, SocketFlags.None) ;
>
> b.Write(read) ;
> totalrecieved += 8192;
> }
>
> string OK = "OK" ;
> byte[] okb = System.Text.Encoding.ASCII.GetBytes(OK) ;
> Console.WriteLine(okb.Length) ;
> sock.Send(okb, 0, okb.Length, SocketFlags.None) ;
> }
>
> b.Close() ;
> MessageBox.Show("DONE") ;
> //f.Close() ;
>
> }
>
>
> I have put the 2 files used at:
>
> http://www.lancs.ac.uk/ug/gloverd/out.mp3 - the file used by the SENDER
> http://www.lancs.ac.uk/ug/gloverd/recieved.mp3 - the output file from
> the reciever. Using a hex comparer, it can be seen that the errors start
> on byte 19519.
>
>
> Many thanks for any help anyone can provide, as I have spent quite a
> while getting nowhere on this problem.
>
> Regards and thanks,
>
> David
>
>
>
> *** Sent via Developersdex http://www.developersdex.com ***
> Don't just participate in USENET...get rewarded for it!

If you're going to handle multiple clients, maybe a multi-threaded approach
would be the best way to go. If a single client wants to grab status
information in the middle, you still have to decide whether to send that
request on the established connection or start a new one just for the status
information, but at least separate clients won't bump into each other.

Paul T.

"David Glover" <(E-Mail Removed)> wrote in message
news:%(E-Mail Removed)...
> Basically the file needs to be segmented to allow a situation where by
> the client (which is on a wireless link) uploads a file, and at the same
> time, the user presses a button such as "get lastest server
> information", which needs to send a message to the server, and then
> recieve a message back. Other wise the user will have to wait for the
> compelete transmission of the file in question before they can initiate
> another operation.
>
> Considering the applicaiton being developed is a distributed network of
> files (using MP3 files in this case), the server get a request from a
> client for a file off another client. The 2nd client then starts to
> upload the file. During this up load the user of the second client
> wishes to see what tracks are availible on the system. The user would be
> blocked by an operation they did not initiate, and this would be poor
> from a usability perspective.
>
> Many thnaks for your input however, I am reluctant to give up this idea
> of segmenting the file considering the operating envrionment. This is
> being a harder nut to crack that 1st thought.
>
>
>
> *** Sent via Developersdex http://www.developersdex.com ***
> Don't just participate in USENET...get rewarded for it!

I see no reason it must be packetized. Just use a worker thread to transfer
the data. You're not limited to a single socket connection (I think it's
128?).

--
Chris Tacke, eMVP
Co-Founder and Advisory Board Member
www.OpenNETCF.org
---
Windows CE Product Manager
Applied Data Systems
www.applieddata.net


"David Glover" <(E-Mail Removed)> wrote in message
news:%(E-Mail Removed)...
> Basically the file needs to be segmented to allow a situation where by
> the client (which is on a wireless link) uploads a file, and at the same
> time, the user presses a button such as "get lastest server
> information", which needs to send a message to the server, and then
> recieve a message back. Other wise the user will have to wait for the
> compelete transmission of the file in question before they can initiate
> another operation.
>
> Considering the applicaiton being developed is a distributed network of
> files (using MP3 files in this case), the server get a request from a
> client for a file off another client. The 2nd client then starts to
> upload the file. During this up load the user of the second client
> wishes to see what tracks are availible on the system. The user would be
> blocked by an operation they did not initiate, and this would be poor
> from a usability perspective.
>
> Many thnaks for your input however, I am reluctant to give up this idea
> of segmenting the file considering the operating envrionment. This is
> being a harder nut to crack that 1st thought.
>
>
>
> *** Sent via Developersdex http://www.developersdex.com ***
> Don't just participate in USENET...get rewarded for it!

I think you may be right!! Looking at the file, everytime the hex values
0D 0A appears, I only get one of them out at the other end! and 0D and
0A refer to carrige return and line feed in ASCII.

I am however still having the problem with the above values when I use
FileStream.write, and again if I use a BinaryWriter with Encoding.UTF-8
set. What could be causing this?? Is there any way to combat this -
other than to process each individual byte at a time in the buffer?

Thanks for your help in tracking down this problem thus far Hugley
appriciated.

David


*** Sent via Developersdex http://www.developersdex.com ***
Don't just participate in USENET...get rewarded for it!

I haven't done any file transfer with .NET CF sockets, but what you really
want is *no* encoding or binary encoding. I'd look around for a way to make
*that* happen.

Paul T.

"David Glover" <(E-Mail Removed)> wrote in message
news:(E-Mail Removed)...
> I think you may be right!! Looking at the file, everytime the hex values
> 0D 0A appears, I only get one of them out at the other end! and 0D and
> 0A refer to carrige return and line feed in ASCII.
>
> I am however still having the problem with the above values when I use
> FileStream.write, and again if I use a BinaryWriter with Encoding.UTF-8
> set. What could be causing this?? Is there any way to combat this -
> other than to process each individual byte at a time in the buffer?
>
> Thanks for your help in tracking down this problem thus far Hugley
> appriciated.
>
> David
>
>
> *** Sent via Developersdex http://www.developersdex.com ***
> Don't just participate in USENET...get rewarded for it!