Using UDP sockets with large chunks of data... is there...?

[ThunderMusic]

Hi,
I'm trying to use a UDP socket between two (or many more) applications
(actually, the same app, but on different machines). For the vast majority
of the messages, it's perfect because it transfers between 10 to 50 bytes.
The protential problem arises when two of these messages can contain
litterally MBs of data... I haven't had the problem yet and haven't been
able to test(achieve what I state next) it, but I'm asking myself the
following question.

Considering I'm using multicast on a local network, would it be possible if
two or more nodes were sending such stuff on the network at the same time
that I would receive their packets in an undetermined order and even receive
the buffer with parcket 1 from A, packet 1 from B, packet 2 from A... and so
on? if it is possible, is there a built-in mecanism to avoid that or will I
have to segment my packets by myself? I would like to avoid TCP at any cost
because there could be 20 to 30 servers talking to each others and multicast
is extremely important to avoid desynchronisation.

Thanks

ThunderMusic

 

[Peter Duniho]

[...]
Considering I'm using multicast on a local network, would it be possible if
two or more nodes were sending such stuff on the network at the same time
that I would receive their packets in an undetermined order and even receive
the buffer with parcket 1 from A, packet 1 from B, packet 2 from A... and so
on?

Yes, it's possible. Likely, even, depending on the configuration of the
network.

if it is possible, is there a built-in mecanism to avoid that or will I
have to segment my packets by myself? I would like to avoid TCP at any cost
because there could be 20 to 30 servers talking to each others and multicast
is extremely important to avoid desynchronisation.

Nevermind the overhead involved in having 20-30 servers each having
20-30 TCP connections to the other servers.

Anyway, no...there's no built-in mechanism to deal with ordering of the
data, or any of the other limitations of UDP. With UDP, any given
datagram may:

* fail to arrive
* arrive more than once
* arrive out of sequence

It is up to the application to handle these issues.

Have you heard of the RTP protocol? It doesn't address these issues,
but I have heard it is supposed to include features that make it easier
for an application to deal with them. I don't have first-hand
experience with it, so I can't offer anything more specific than that.
You can read more here:
http://www.faqs.org/rfcs/rfc1889.html

Your post implies that keeping all of the recipients of the data
synchronized is important, so you are likely going to have to just deal
with the problems inherent in UDP. However, if for some reason I've
misunderstood, one possible solution would be to use TCP in a sort of
"ring topology", where each instance of your program connects with only
two other instances, receiving from one and sending to the other. Any
data received would be sent along to the other, as well as being
processed locally of course.

In this way, the network overhead would be much less than that required
for an N-way connection implementation. Of course, the downside is that
the latency between the first instance in the chain and the last
instance in the chain could be huge. They definitely won't be
synchronized. But if that's not important, it could be simpler to
implement than dealing with the datagram problems of UDP.

Pete

 

[ThunderMusic]

[...]
Considering I'm using multicast on a local network, would it be possible
if two or more nodes were sending such stuff on the network at the same
time that I would receive their packets in an undetermined order and even
receive the buffer with parcket 1 from A, packet 1 from B, packet 2 from
A... and so on?

Yes, it's possible. Likely, even, depending on the configuration of the
network.

if it is possible, is there a built-in mecanism to avoid that or will I
have to segment my packets by myself? I would like to avoid TCP at any
cost because there could be 20 to 30 servers talking to each others and
multicast is extremely important to avoid desynchronisation.

Nevermind the overhead involved in having 20-30 servers each having 20-30
TCP connections to the other servers.

Anyway, no...there's no built-in mechanism to deal with ordering of the
data, or any of the other limitations of UDP. With UDP, any given
datagram may:

* fail to arrive
* arrive more than once
* arrive out of sequence

It is up to the application to handle these issues.

Have you heard of the RTP protocol? It doesn't address these issues, but
I have heard it is supposed to include features that make it easier for an
application to deal with them. I don't have first-hand experience with
it, so I can't offer anything more specific than that. You can read more
here:
http://www.faqs.org/rfcs/rfc1889.html

Your post implies that keeping all of the recipients of the data
synchronized is important, so you are likely going to have to just deal
with the problems inherent in UDP. However, if for some reason I've
misunderstood, one possible solution would be to use TCP in a sort of
"ring topology", where each instance of your program connects with only
two other instances, receiving from one and sending to the other. Any
data received would be sent along to the other, as well as being processed
locally of course.

In this way, the network overhead would be much less than that required
for an N-way connection implementation. Of course, the downside is that
the latency between the first instance in the chain and the last instance
in the chain could be huge. They definitely won't be synchronized. But
if that's not important, it could be simpler to implement than dealing
with the datagram problems of UDP.

Pete

hi,
The synchronisation is a crutial part of the software so I can't do without
it.

Thanks a lot for the information. One more question: Is there a maximum
length for a UDP packet? Using .NET, how can I make sure I don't bust this
maximum per packet so I can arrange the data in such a manner I will be able
to reconstitute my whole message at the end and recall a packet if it's
missing? Because we are not aware of the headers and footers of packets when
using the .NET Socket class.

Thanks a lot...

ThunderMusic

 

[Peter Duniho]

[...]
Thanks a lot for the information. One more question: Is there a maximum
length for a UDP packet?

Yes. There are actually at least two relevant limits. One is the
fundamental maximum size of a UDP datagram, which is just under 64K.
The other, probably more important to you, is the "maximum transmission
unit" for your network.

A typical MTU is 1500 bytes. If you send UDP datagrams larger than
this, then the datagram will be fragmented into smaller pieces and
reassembled at the other end. The main issue is that if any of that
smaller pieces is lost, you lose the entire datagram.

By ensuring that each of your datagrams is no larger than the MTU, then
if a network packet is dropped, you've only lost that one packet of
data, rather than potentially a lot more (for example, in the worst case
using 64K UDP datagrams, losing a single 1500 byte packet would cause
you to lose nearly 63K of data that was not necessarily otherwise affected).

Using .NET, how can I make sure I don't bust this
maximum per packet so I can arrange the data in such a manner I will be able
to reconstitute my whole message at the end and recall a packet if it's
missing? Because we are not aware of the headers and footers of packets when
using the .NET Socket class.

Just make sure your datagrams are small enough. .NET doesn't provide
details of the UDP headers, but those are standardized. So just
subtract the size of the UDP header from the MTU and use that as your
limit for your datagram size.

Pete

 

[ThunderMusic]

Thanks a lot... That really helps... and I found an article on this very
topic on Wikipedia (I always forget to look there first)

http://en.wikipedia.org/wiki/User_Datagram_Protocol

Thanks

ThunderMusic

[...]
Thanks a lot for the information. One more question: Is there a maximum
length for a UDP packet?

Yes. There are actually at least two relevant limits. One is the
fundamental maximum size of a UDP datagram, which is just under 64K. The
other, probably more important to you, is the "maximum transmission unit"
for your network.

A typical MTU is 1500 bytes. If you send UDP datagrams larger than this,
then the datagram will be fragmented into smaller pieces and reassembled
at the other end. The main issue is that if any of that smaller pieces is
lost, you lose the entire datagram.

By ensuring that each of your datagrams is no larger than the MTU, then if
a network packet is dropped, you've only lost that one packet of data,
rather than potentially a lot more (for example, in the worst case using
64K UDP datagrams, losing a single 1500 byte packet would cause you to
lose nearly 63K of data that was not necessarily otherwise affected).

Using .NET, how can I make sure I don't bust this maximum per packet so I
can arrange the data in such a manner I will be able to reconstitute my
whole message at the end and recall a packet if it's missing? Because we
are not aware of the headers and footers of packets when using the .NET
Socket class.

Just make sure your datagrams are small enough. .NET doesn't provide
details of the UDP headers, but those are standardized. So just subtract
the size of the UDP header from the MTU and use that as your limit for
your datagram size.

Pete

 

[ThunderMusic]

Is there a way to find the actual MTU of the networks adapter(s) from .NET
or will I have to add a new setting to my application?

Thanks

ThunderMusic

[...]
Thanks a lot for the information. One more question: Is there a maximum
length for a UDP packet?

Yes. There are actually at least two relevant limits. One is the
fundamental maximum size of a UDP datagram, which is just under 64K. The
other, probably more important to you, is the "maximum transmission unit"
for your network.

A typical MTU is 1500 bytes. If you send UDP datagrams larger than this,
then the datagram will be fragmented into smaller pieces and reassembled
at the other end. The main issue is that if any of that smaller pieces is
lost, you lose the entire datagram.

By ensuring that each of your datagrams is no larger than the MTU, then if
a network packet is dropped, you've only lost that one packet of data,
rather than potentially a lot more (for example, in the worst case using
64K UDP datagrams, losing a single 1500 byte packet would cause you to
lose nearly 63K of data that was not necessarily otherwise affected).

Using .NET, how can I make sure I don't bust this maximum per packet so I
can arrange the data in such a manner I will be able to reconstitute my
whole message at the end and recall a packet if it's missing? Because we
are not aware of the headers and footers of packets when using the .NET
Socket class.

Just make sure your datagrams are small enough. .NET doesn't provide
details of the UDP headers, but those are standardized. So just subtract
the size of the UDP header from the MTU and use that as your limit for
your datagram size.

Pete

 

[Scott Gifford]

Is there a way to find the actual MTU of the networks adapter(s) from .NET
or will I have to add a new setting to my application?

In general, to find the MTU between two Internet hosts, the technique
for finding the largest MTU between them is called "Path MTU
Discovery", or PMTUD.

The Wikipedia article contains some information and links to the
relevant RFCs:

http://en.wikipedia.org/wiki/Path_MTU_discovery

PMTUD is a hassle and uses a bit of bandwidth, so most applications
just assume it will be a few bytes smaller than Ethernet MTU of 1500
bytes, to make room for headers. You can probably find out more in
comp.protocols.tcp-ip if you decide to implement this.

If you know the host are all on the same broadcast network (won't be
going through any routers), you should be able to get the MTU as a
property of the network interface, though I don't recall exactly how.

Good luck!

----Scott.

 

[ThunderMusic]

Hi,
It definitly can be useful... right now, I was thinking more about getting
the value from the network interface which is done using the
System.Net.NetworkInformation.IPv4InterfaceProperties class. It would be
more to give a warning tought... because the user could configure it the way
he wants, but he'll be warned that he may observe unexpected behaviors.

This is the link to the documentation of the stated class:
http://msdn2.microsoft.com/en-us/li...formation.ipv4interfaceproperties(vs.80).aspx

It may also be important to note (if someone finds this thread and is
looking for answers) that the default MTU size changes depending on the
network type as described in this article:
http://support.microsoft.com/kb/140375

Thanks a lot to all for your help...

ThunderMusic