Handling severe packet loss (serialization).

J

jkv

Hi,

I am developing a application which have to continue working even in a
high packet loss environment, i have attached the serialization and
network transmission parts from both the server and client.
It works when there are no packet loss, but when introducing severe
packet loss (10%) it starts crashing because the ' int size =
br.ReadInt32();' in the server snippets reads some random data.

My thoughts on how to fix this is to introduce a uniq prefix and
postfix on ms.ToArray().Length(fx. ???!!!512!!!???) and send this
string instead of just 'bw.Write(ms.ToArray().Length);'.

But how would i make the server catch this "delimited" string? The
idea is to let the server skip data until it reaches the delimited
string and then afterwards start deserialization as normal, and handle
any deserialization faults with try/catch.

I'm fully aware the the above solution would very likely miss alot of
objects - but that dosent really matter.

Any hints on a the above - or anyone got a idea for an entirely
differnt approach?

Regards,
Johnny

---CLIENT SNIPPET--

private void Sendchannel()
{
BinaryWriter bw = new BinaryWriter(_clientStream);

Int32 sizeOfPacket;

while (!SharedObjects.ShuttingDown)
{
while (SharedObjects.clientOutgoingQueue.Count !=
0)
{
Common.SIO obj =
SharedObjects.clientOutgoingQueue.Dequeue();
MemoryStream ms = new MemoryStream();
IFormatter formatter = new BinaryFormatter();
formatter.Serialize(ms, obj);
bw.Write(ms.ToArray().Length);
_clientStream.Flush();
bw.Write(ms.ToArray());
_clientStream.Flush();

}
Thread.Sleep(100);
}
}





---SERVER SNIPPET---

private void receiveChannel(NetworkStream clientstream)
{
Int32 sizeOfPacket;

BinaryReader br = new BinaryReader(clientstream);
BinaryWriter bw = new BinaryWriter(clientstream);


while (!SharedObjects.ShuttingDown)
{
if (clientstream.DataAvailable)
{
byte[] message = new byte[4096];
int size = br.ReadInt32();
MemoryStream ms = new MemoryStream();
message = br.ReadBytes(size);
ms.Write(message, 0, message.Length);
IFormatter formatter = new BinaryFormatter();
ms.Seek(0, SeekOrigin.Begin);
Common.SIO testsio =
(Common.SIO)formatter.Deserialize(ms);
SharedObjects.IncommingQueue.Enqueue(testsio);

}
else
{
Thread.Sleep(100);
}
}
}
 
N

not_a_commie

Most protocols use a checksum to avoid this problem. Look around on
the web at CRC32 implementations. I think you could work something
like that in to your serialization code.
 
P

Peter Duniho

jkv said:
Hi,

I am developing a application which have to continue working even in a
high packet loss environment, i have attached the serialization and
network transmission parts from both the server and client.
It works when there are no packet loss, but when introducing severe
packet loss (10%) it starts crashing because the ' int size =
br.ReadInt32();' in the server snippets reads some random data. [...]

What protocol are you using? TCP shouldn't have any trouble. If you're
using UDP, switch to TCP or don't break your data across datagrams. If
you're using TCP, then you've done something wrong, probably by making
the incorrect assumption that the data will be delivered in the same
groups in which it was sent.

Unfortunately, what little code you did post didn't show any of the
actual network-related implementation details. It's not really possible
to understand from your post why the "high packet loss environment" is
causing trouble. But most likely, the answer will be: "use TCP, and use
it correctly".

Pete
 
M

Mr. Arnold

jkv said:
Any hints on a the above - or anyone got a idea for an entirely
differnt approach?

If you're using .NET Framework 2.0 or better, you could use WCF.

<http://www.request-response.com/blog/PermaLink,guid,a6cb59e6-cbc2-4ce3-92b2-ea40bc5929f6.aspx>

You can use WCF over TCP/IP WCF cleint and WCF service.

You can use WCF over HTTP Web service.

You can use WCF over TCP/IP service.

You can use WCF over Named Pipe service.

You can use WCF over MSMQ service.

WCF does the serialization and deserialization. It takes care of
everything in packet delivery between the client and service, so
developer (you) no longer needs to be concerned about it.

It's MS's preferred means of client and service communications, with new
technology geared towards using WCF technology, which is really simple
to use when it come right down to it.

The code you posted is using old legacy technology -- none WCF and
obsolete.

You should get yourself a good book on WCF.
 
J

jkv

Hi,

jkv wrote:
What protocol are you using?  TCP shouldn't have any trouble.  If you're
using UDP, switch to TCP or don't break your data across datagrams.  If
you're using TCP, then you've done something wrong, probably by making
the incorrect assumption that the data will be delivered in the same
groups in which it was sent.

You are right, i was assuming that TcpClient.getstream() ensured that
packets arrived in order,
but when i read your posting i get the feeling that this is not the
case :)
Could you elaborate on this?
Unfortunately, what little code you did post didn't show any of the
actual network-related implementation details.  It's not really possible
to understand from your post why the "high packet loss environment" is
causing trouble.  But most likely, the answer will be: "use TCP, and use
it correctly".

I am using TCP, and as far as i know i use it correctly.

The clientstream's in the client methods i posted earlier i from:

private TcpClient tcpClient;
IPEndPoint serverEndPoint = new
IPEndPoint(IPAddress.Parse("192.168.0.1"), port);
tcpClient = new TcpClient();
tcpClient.Connect(serverEndPoint);
_clientStream = tcpClient.GetStream();
Sendchannel()

and the _clientstream in the server methods is from:

sendListener = new
TcpListener(IPAddress.Any, 3001);
sendListener.Start();
TcpClient client =
this.receiveListener.AcceptTcpClient();
Thread clientThread = new
Thread(HandleClientComm);
clientThread.Start(client);

Regards,
Johnny
 
J

jkv

Hi,

jkv wrote:
What protocol are you using?  TCP shouldn't have any trouble.  If you're
using UDP, switch to TCP or don't break your data across datagrams.  If
you're using TCP, then you've done something wrong, probably by making
the incorrect assumption that the data will be delivered in the same
groups in which it was sent.

You are right, i was assuming that TcpClient.getstream() ensured that
packets arrived in order,
but when i read your posting i get the feeling that this is not the
case :)
Could you elaborate on this?
Unfortunately, what little code you did post didn't show any of the
actual network-related implementation details.  It's not really possible
to understand from your post why the "high packet loss environment" is
causing trouble.  But most likely, the answer will be: "use TCP, and use
it correctly".

I am using TCP, and as far as i know i use it correctly.

The clientstream's in the client methods i posted earlier i from:

private TcpClient tcpClient;
IPEndPoint serverEndPoint = new
IPEndPoint(IPAddress.Parse("192.168.0.1"), port);
tcpClient = new TcpClient();
tcpClient.Connect(serverEndPoint);
_clientStream = tcpClient.GetStream();
Sendchannel()

and the _clientstream in the server methods is from:

sendListener = new
TcpListener(IPAddress.Any, 3001);
sendListener.Start();
TcpClient client =
this.receiveListener.AcceptTcpClient();
Thread clientThread = new
Thread(HandleClientComm);
clientThread.Start(client);

Regards,
Johnny
 
J

jkv

Hi,

jkv wrote:
What protocol are you using?  TCP shouldn't have any trouble.  If you're
using UDP, switch to TCP or don't break your data across datagrams.  If
you're using TCP, then you've done something wrong, probably by making
the incorrect assumption that the data will be delivered in the same
groups in which it was sent.

You are right, i was assuming that TcpClient.getstream() ensured that
packets arrived in order,
but when i read your posting i get the feeling that this is not the
case :)
Could you elaborate on this?
Unfortunately, what little code you did post didn't show any of the
actual network-related implementation details.  It's not really possible
to understand from your post why the "high packet loss environment" is
causing trouble.  But most likely, the answer will be: "use TCP, and use
it correctly".

I am using TCP, and as far as i know i use it correctly.

The clientstream's in the client methods i posted earlier i from:

private TcpClient tcpClient;
IPEndPoint serverEndPoint = new
IPEndPoint(IPAddress.Parse("192.168.0.1"), port);
tcpClient = new TcpClient();
tcpClient.Connect(serverEndPoint);
_clientStream = tcpClient.GetStream();
Sendchannel()

and the _clientstream in the server methods is from:

sendListener = new
TcpListener(IPAddress.Any, 3001);
sendListener.Start();
TcpClient client =
this.receiveListener.AcceptTcpClient();
Thread clientThread = new
Thread(HandleClientComm);
clientThread.Start(client);

Regards,
Johnny
 
J

jkv

Hi,

jkv wrote:
What protocol are you using?  TCP shouldn't have any trouble.  If you're
using UDP, switch to TCP or don't break your data across datagrams.  If
you're using TCP, then you've done something wrong, probably by making
the incorrect assumption that the data will be delivered in the same
groups in which it was sent.

You are right, i was assuming that TcpClient.getstream() ensured that
packets arrived in order,
but when i read your posting i get the feeling that this is not the
case :)
Could you elaborate on this?
Unfortunately, what little code you did post didn't show any of the
actual network-related implementation details.  It's not really possible
to understand from your post why the "high packet loss environment" is
causing trouble.  But most likely, the answer will be: "use TCP, and use
it correctly".

I am using TCP, and as far as i know i use it correctly.

The clientstream's in the client methods i posted earlier i from:

private TcpClient tcpClient;
IPEndPoint serverEndPoint = new
IPEndPoint(IPAddress.Parse("192.168.0.1"), port);
tcpClient = new TcpClient();
tcpClient.Connect(serverEndPoint);
_clientStream = tcpClient.GetStream();
Sendchannel()

and the _clientstream in the server methods is from:

sendListener = new
TcpListener(IPAddress.Any, 3001);
sendListener.Start();
TcpClient client =
this.receiveListener.AcceptTcpClient();
Thread clientThread = new
Thread(HandleClientComm);
clientThread.Start(client);

Regards,
Johnny
 
P

Peter Duniho

jkv said:
Hi,




You are right, i was assuming that TcpClient.getstream() ensured that
packets arrived in order,
but when i read your posting i get the feeling that this is not the
case :)
Could you elaborate on this?

TCP does ensure that the "packets" arrive in order. But TCP doesn't
care about "packets". So the _bytes_ are in the same order in which
they are sent, but the boundaries between groups of bytes can be
completely different.

Any given call to receive from a TCP socket may return anywhere between
a single byte and the size of the buffer you pass it. Your own code has
to properly handle reassembling those bytes into "packets" if that
indeed is the foundation of your application-level protocol.

See the Winsock FAQ for more details. In spite of being
Winsock-specific, there's a lot of information in there that applies to
network programming generally, and of course since the .NET Socket API
is based on Winsock, even the Winsock-specific information can be useful
to know.

http://tangentsoft.net/wskfaq/
I am using TCP, and as far as i know i use it correctly.

But isn't that true for all of us, for whatever API we're using, until
we start having problems?

Looking more closely at the code you posted in your first message, I can
already see that you aren't using TCP correctly, because you're polling
the data stream.

On the other hand, using the BinaryReader should ensure that .NET takes
care of reassembling the data as it was sent. There's definitely some
inefficiencies in the code you posted, but I don't see anything obvious
that would prevent it from working.

On the other hand, you say it's not working. And if you're using TCP,
network congestion is not going to cause you to lose or corrupt data.
So, unless your network itself is corrupting the data (not very likely)
that means _something_ about your code is in fact not correct.

Unfortunately, without a concise-but-complete code example that reliably
demonstrates the problem, it's not really possible to know for sure what
the problem might be. What I can assure you of is that TCP doesn't
normally drop or corrupt your data, even under extreme network congestion.

Pete
 
J

jkv

Hi,

But isn't that true for all of us, for whatever API we're using, until
we start having problems?

Looking more closely at the code you posted in your first message, I can
already see that you aren't using TCP correctly, because you're polling
the data stream.

You are right, i misinterpreted how Networkstream.Read() works, but
after several hints
from posters i finally got it right by looping NetworkStream.Read()
until all expected bytes are read.

Thanks to all for the replies regarding this, and also a thank to 'Mr.
Arnold' for poiting out WCF as a possible solution, i will be looking
info WFC whenever i get around to it.

Regards,
Johnny
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Top