receiving data before Socket.EndReceive()

[darthghandi]

I am having mixed results with asynchronous socket receives.
Sometimes I get the right information back from the buffer, other
times I get some of the data that should be in the buffer printed out
to the console. That data is printed out to the console before I even
call Socket.EndReceive(). After that call, I get the rest of the data
that wasn't printed out to the console in the buffer. Anyone have
any idea what is going on?
Here's the code:
public delegate void ReceivedPacketCallBack(Packet pack);
class PacketReceiver
{
private ReceivedPacketCallBack m_newPacket;
private IOBuffer m_socketAndBuffer;
private int m_blockSize;
private uint m_packetSize;
private uint m_receivedBytes;
private List<byte> m_holdBuffer;

public PacketReceiver(IOBuffer theSockBuff, int blockSize)
{
m_socketAndBuffer = theSockBuff;
m_blockSize = blockSize;
m_socketAndBuffer.ClearBuffer();
m_packetSize = 0;
m_receivedBytes = 0;
//m_holdBuffer = new List<byte>();
}

public void NeedPacket(ReceivedPacketCallBack pack)
{
m_newPacket = pack;
}

//should make sure that we receive one and only one packet
public void ReceivePacket()
{
m_socketAndBuffer.ClearBuffer();
if (m_socketAndBuffer.IsConnected)
{
AsyncCallback callMe = new
AsyncCallback(ReceiveCallBack);
m_socketAndBuffer.ResizeByteBuffer((uint)m_blockSize);
Console.WriteLine("The Buffer is " +
m_socketAndBuffer.m_buffer.Count + " big in bytes.");

m_socketAndBuffer.m_sockMe.BeginReceive(m_socketAndBuffer.m_buffer,
SocketFlags.None, callMe, m_socketAndBuffer);
}
else
{
throw new Exception("The socket is no longer
connected.");
}
}

private void ReceiveCallBack(IAsyncResult ar)
{
m_socketAndBuffer = (IOBuffer)ar.AsyncState;
if (m_socketAndBuffer.IsConnected)
{
m_receivedBytes =
(uint)m_socketAndBuffer.m_sockMe.EndReceive(ar);
m_holdBuffer = new List<byte>((int)m_receivedBytes);

m_holdBuffer.AddRange(m_socketAndBuffer.m_buffer[0].Array);
m_socketAndBuffer.ClearBuffer();
m_packetSize = Packet.ReadInt32FromBytes(m_holdBuffer,
0);
Console.WriteLine("The packet size is " + m_packetSize
+ "");
UInt32 bytesLeft = m_packetSize - (uint)(m_blockSize -
4);
Console.WriteLine("We need to read " + bytesLeft + "
more bytes.");
if (bytesLeft > 512000000)
{
Console.WriteLine("size to big");
}
m_socketAndBuffer.ResizeByteBuffer(bytesLeft);

AsyncCallback callme = new
AsyncCallback(CreatePacket);

m_socketAndBuffer.m_sockMe.BeginReceive(m_socketAndBuffer.m_buffer,
SocketFlags.None, callme, m_socketAndBuffer);
}
}

private void CreatePacket(IAsyncResult ar)
{
m_socketAndBuffer = (IOBuffer)ar.AsyncState;
if (m_socketAndBuffer.IsConnected)
{
m_receivedBytes +=
(uint)m_socketAndBuffer.m_sockMe.EndReceive(ar);

m_holdBuffer.AddRange(m_socketAndBuffer.m_buffer[0].Array);
if (m_receivedBytes < (m_packetSize + 4))
{
uint leftOver = m_receivedBytes - m_packetSize +
4;
m_socketAndBuffer.ResizeByteBuffer(leftOver);
AsyncCallback callMeNow = new
AsyncCallback(CreatePacket);

m_socketAndBuffer.m_sockMe.BeginReceive(m_socketAndBuffer.m_buffer,
SocketFlags.None, callMeNow, m_socketAndBuffer);
}
Packet thePacket = Packet.ParseBytes(m_holdBuffer,
m_blockSize);
m_newPacket(thePacket); //send the packet to
whoever needs it (delegate)
}
}
}

 

[Peter Duniho]

I am having mixed results with asynchronous socket receives.
Sometimes I get the right information back from the buffer, other
times I get some of the data that should be in the buffer printed out
to the console. That data is printed out to the console before I even
call Socket.EndReceive(). After that call, I get the rest of the data
that wasn't printed out to the console in the buffer. Anyone have
any idea what is going on?
Here's the code:

That is both not enough code and too much, all at the same time. You
are missing half of the connection (the sending side) as well as the
implementation of IOBuffer, and as well you have included a fair amount
of code that is likely not necessary to reproduce the issue (in fact,
unless IOBuffer is directly related to the problem, you should post code
that doesn't even include it).

And you have not precisely described what the actual problem is. What
output do you get, and what output do you expect instead?

So, if you really want someone to spend more than a few moments with
your code, you should distill it down into something that is
concise-but-complete while reliably reproducing the problem. Post that
along with a precise problem description.

That said, as near as I can tell from my quick scan of your code, you
appear to deliver an assembled "packet" whether it's completed or not.
I would expect that to be a problem.

There are plenty of other things I would change about the code, but that
stands out as being particularly likely to produce incorrect results.

Pete

 

[darthghandi]

That is both not enough code and too much, all at the same time. You
are missing half of the connection (the sending side) as well as the
implementation of IOBuffer, and as well you have included a fair amount
of code that is likely not necessary to reproduce the issue (in fact,
unless IOBuffer is directly related to the problem, you should post code
that doesn't even include it).

And you have not precisely described what the actual problem is. What
output do you get, and what output do you expect instead?

So, if you really want someone to spend more than a few moments with
your code, you should distill it down into something that is
concise-but-complete while reliably reproducing the problem. Post that
along with a precise problem description.

That said, as near as I can tell from my quick scan of your code, you
appear to deliver an assembled "packet" whether it's completed or not.
I would expect that to be a problem.

There are plenty of other things I would change about the code, but that
stands out as being particularly likely to produce incorrect results.

Pete

Here is the section of code where I use the PacketReceiver:
private void EndStart(IAsyncResult ar)
{
Socket temp = (Socket)ar.AsyncState;
if (temp.Connected)
{
int sentBytes = temp.EndSend(ar);
List<ArraySegment<byte>> tempBuffer = new
List<ArraySegment<byte>>();
tempBuffer.Add(new ArraySegment<byte>(new byte[255]));
receiveBuffer.m_buffer = tempBuffer;
AsyncCallback callMe = new
AsyncCallback(EndReceiveVersionString);

receiveBuffer.m_sockMe.BeginReceive(receiveBuffer.m_buffer,
SocketFlags.None, callMe, receiveBuffer);
}
}

private void EndReceiveVersion(IAsyncResult ar)
{
Encoder theEncoder = new Encoder();
IOBuffer temp = (IOBuffer)ar.AsyncState;
if (temp.m_sockMe.Connected)
{
int receivedBytes = temp.m_sockMe.EndReceive(ar);

Packet sendAlgs = new Packet();
sendAlgs.Fill(sendData);
ArraySegment<byte> toBuff = new
ArraySegment<byte>(sendAlgs.GetPacketInBytes());
sendBuffer.m_buffer.Add(toBuff);
sendBuffer.m_sockMe.BeginSend(sendBuffer.m_buffer,
SocketFlags.None, callMe, sendBuffer);
}
}
I have not experienced problems sending, just receiving. The output I
am seeing in the console is data sent from the client, but only the
first part of it. The rest of the data sent from the client is in the
buffer like it should be. The packet is assembled by first reading
the length (which should be the first of the packet), then doing
another receive call to get the rest of the packet. Since some of the
data received is sent to the console instead of the buffer, I get a
bad length when I try to do the second receive call. For example, if
the client sends a packet like this:
23somedata
the console would display the following (which it shouldn't):
23som
and the buffer would contain:
edata
Does that make sense?
Thanks for your time.

 

[darthghandi]

That is both not enough code and too much, all at the same time. You
are missing half of the connection (the sending side) as well as the
implementation of IOBuffer, and as well you have included a fair amount
of code that is likely not necessary to reproduce the issue (in fact,
unless IOBuffer is directly related to the problem, you should post code
that doesn't even include it).

And you have not precisely described what the actual problem is. What
output do you get, and what output do you expect instead?

So, if you really want someone to spend more than a few moments with
your code, you should distill it down into something that is
concise-but-complete while reliably reproducing the problem. Post that
along with a precise problem description.

That said, as near as I can tell from my quick scan of your code, you
appear to deliver an assembled "packet" whether it's completed or not.
I would expect that to be a problem.

There are plenty of other things I would change about the code, but that
stands out as being particularly likely to produce incorrect results.

Pete

Here is the section of code where I use the PacketReceiver:
private void EndStart(IAsyncResult ar)
{
Socket temp = (Socket)ar.AsyncState;
if (temp.Connected)
{
int sentBytes = temp.EndSend(ar);
List<ArraySegment<byte>> tempBuffer = new
List<ArraySegment<byte>>();
tempBuffer.Add(new ArraySegment<byte>(new byte[255]));
receiveBuffer.m_buffer = tempBuffer;
AsyncCallback callMe = new
AsyncCallback(EndReceiveVersionString);

receiveBuffer.m_sockMe.BeginReceive(receiveBuffer.m_buffer,
SocketFlags.None, callMe, receiveBuffer);
}
}

private void EndReceiveVersion(IAsyncResult ar)
{
Encoder theEncoder = new Encoder();
IOBuffer temp = (IOBuffer)ar.AsyncState;
if (temp.m_sockMe.Connected)
{
int receivedBytes = temp.m_sockMe.EndReceive(ar);

Packet sendAlgs = new Packet();
sendAlgs.Fill(sendData);
ArraySegment<byte> toBuff = new
ArraySegment<byte>(sendAlgs.GetPacketInBytes());
sendBuffer.m_buffer.Add(toBuff);
sendBuffer.m_sockMe.BeginSend(sendBuffer.m_buffer,
SocketFlags.None, callMe, sendBuffer);
}
}
I have not experienced problems sending, just receiving. The output I
am seeing in the console is data sent from the client, but only the
first part of it. The rest of the data sent from the client is in the
buffer like it should be. The packet is assembled by first reading
the length (which should be the first of the packet), then doing
another receive call to get the rest of the packet. Since some of the
data received is sent to the console instead of the buffer, I get a
bad length when I try to do the second receive call. For example, if
the client sends a packet like this:
23somedata
the console would display the following (which it shouldn't):
23som
and the buffer would contain:
edata
Does that make sense?
Thanks for your time.

Also, please feel free to make suggestions on how I can improve my
code. I'm really new to this and would appreciate any tips anyone
might have. If there are any good books on sockets or writing good
code please feel free to suggest them.
Thanks for your time.

 

[Peter Duniho]

Here is the section of code where I use the PacketReceiver:

Please re-read the part in my previous post where I discussed the idea
of "concise-but-complete". You don't appear to have done anything at
all to reduce your code to the bare minimum required to reproduce the
problem.

[...]
I have not experienced problems sending, just receiving.

Receiving doesn't happen without sending. Until you have figured out
what your problem is, you don't actually know whether it's a problem
with the sending or the receiving.

The output I
am seeing in the console is data sent from the client, but only the
first part of it. The rest of the data sent from the client is in the
buffer like it should be.

I don't understand this part. You didn't post any code that appears to
display any data at all to the console. The only writes to the console
I see are informational things (buffer size, "packet" size, initial
bytes remaining).

The packet is assembled by first reading
the length (which should be the first of the packet), then doing
another receive call to get the rest of the packet.

As I mentioned in my previous post, you do not appear to be waiting for
a completed "packet" before handing it to your "m_newPacket" delegate.
Whether this is the cause of your troubles, I don't know. But it
certainly appears to be a cause of _some_ troubles.

Since some of the
data received is sent to the console instead of the buffer, I get a
bad length when I try to do the second receive call.

Again with the "sent to the console". Where is it sent to the console?
How have you determined that it's sent to the console?

For example, if
the client sends a packet like this:
23somedata
the console would display the following (which it shouldn't):
23som
and the buffer would contain:
edata
Does that make sense?

No, it makes no sense at all. .NET isn't that random. Since you don't
seem to have any code that writes the received data to the console, I
don't see any way for the received data to be written to the console.
This means either the console isn't really displaying what you say it
is, or you haven't posted all of the relevant code.

Until you post a concise-but-complete sample of code that reliably
reproduces the problem, I am unable to help you. Maybe there's someone
else out there with better intuition than I have who will be able to
guess what the problem is without a suitable example of code to work with.

Please note that, as I mentioned before, "concise-but-complete" means
that you have removed everything from the code that is not absolutely
required in order to reproduce the problem. Don't just post all of your
code and expect someone to review the entire thing. And don't overlook
the "complete" aspect: it is very important that someone can take your
code, copy it to a file, and compile it directly, without any extra
work. Especially with a console application, this should be very simple
to do.

Pete

 

[darthghandi]

Here is the section of code where I use the PacketReceiver:

Please re-read the part in my previous post where I discussed the idea
of "concise-but-complete". You don't appear to have done anything at
all to reduce your code to the bare minimum required to reproduce the
problem.

[...]
I have not experienced problems sending, just receiving.

Receiving doesn't happen without sending. Until you have figured out
what your problem is, you don't actually know whether it's a problem
with the sending or the receiving.

The output I
am seeing in the console is data sent from the client, but only the
first part of it. The rest of the data sent from the client is in the
buffer like it should be.

I don't understand this part. You didn't post any code that appears to
display any data at all to the console. The only writes to the console
I see are informational things (buffer size, "packet" size, initial
bytes remaining).

The packet is assembled by first reading
the length (which should be the first of the packet), then doing
another receive call to get the rest of the packet.

As I mentioned in my previous post, you do not appear to be waiting for
a completed "packet" before handing it to your "m_newPacket" delegate.
Whether this is the cause of your troubles, I don't know. But it
certainly appears to be a cause of _some_ troubles.

Since some of the
data received is sent to the console instead of the buffer, I get a
bad length when I try to do the second receive call.

Again with the "sent to the console". Where is it sent to the console?
How have you determined that it's sent to the console?

For example, if
the client sends a packet like this:
23somedata
the console would display the following (which it shouldn't):
23som
and the buffer would contain:
edata
Does that make sense?

No, it makes no sense at all. .NET isn't that random. Since you don't
seem to have any code that writes the received data to the console, I
don't see any way for the received data to be written to the console.
This means either the console isn't really displaying what you say it
is, or you haven't posted all of the relevant code.

Until you post a concise-but-complete sample of code that reliably
reproduces the problem, I am unable to help you. Maybe there's someone
else out there with better intuition than I have who will be able to
guess what the problem is without a suitable example of code to work with.

Please note that, as I mentioned before, "concise-but-complete" means
that you have removed everything from the code that is not absolutely
required in order to reproduce the problem. Don't just post all of your
code and expect someone to review the entire thing. And don't overlook
the "complete" aspect: it is very important that someone can take your
code, copy it to a file, and compile it directly, without any extra
work. Especially with a console application, this should be very simple
to do.

Pete

I figured part of it out. I was allocating a 255 byte buffer to read
the version from the client. This buffer would sometimes contain more
info than I really wanted, so I just threw it away without bothering
to check if I may need it later. That was the stuff I was seeing
outputted to the console (I still have no clue why the info was
showing up in the console, I didn't put it there). Now comes the hard
part, the redesign. Would it be better to read one byte at a time
looking for line feed or read chunks looking for the same thing and
saving the rest? If anyone knows of some good examples or articles on
good ways to design this type of thing please feel free to suggest
them. Thanks

 

[Peter Duniho]

[...]
Now comes the hard
part, the redesign. Would it be better to read one byte at a time
looking for line feed or read chunks looking for the same thing and
saving the rest? If anyone knows of some good examples or articles on
good ways to design this type of thing please feel free to suggest
them.

Generally speaking, you should receive larger blocks at a time, to make
most efficient use of the network i/o. Then you can parcel that data
out as needed to the actual application end of things.

As another suggestion regarding your code: IMHO, it's not a good idea to
have two different receive callbacks. I understand that you have
different things to do when receiving the data, and that's what led to
the two different callbacks. But your network i/o should be more
separated from the application logic than that (see above).

Even once you get things separated better, IMHO it will still make more
sense to have a general-purpose application data-handling method, but at
least if you don't do it that way, your actual network i/o code remains
simple.

Pete

 

[darthghandi]

[...]
Now comes the hard
part, the redesign. Would it be better to read one byte at a time
looking for line feed or read chunks looking for the same thing and
saving the rest? If anyone knows of some good examples or articles on
good ways to design this type of thing please feel free to suggest
them.

Generally speaking, you should receive larger blocks at a time, to make
most efficient use of the network i/o. Then you can parcel that data
out as needed to the actual application end of things.

As another suggestion regarding your code: IMHO, it's not a good idea to
have two different receive callbacks. I understand that you have
different things to do when receiving the data, and that's what led to
the two different callbacks. But your network i/o should be more
separated from the application logic than that (see above).

Even once you get things separated better, IMHO it will still make more
sense to have a general-purpose application data-handling method, but at
least if you don't do it that way, your actual network i/o code remains
simple.

Pete

I'll give that a shot. Thanks again for your time.

 

[Chris Mullins [MVP]]

You're trying to do the same sorts of things we do in our XMPP SDK.

We read from the various socket streams, parse out XML Fragments, turn them
into packets, match them against classes, instantiate the right object,
serialize the xml into the object, and then raise the correct event.

Our SDK has source code available, and you can pull it down and look at how
we're doing everything:
http://developers.coversant.net/Downloads/tabid/53/Default.aspx

What you're looking for is (mostly) found in the two classes:
Coversant.SoapBox.Base.XMPPSocket
Coversant.SoapBox.Base.XMLStreamReceiver

Our approach is very scalable, and quite reliable. Our use of socket buffers
that comes out of a buffer pool for reads/write, minimizes heap
fragmentation.

 

[darthghandi]

You're trying to do the same sorts of things we do in our XMPP SDK.

We read from the various socket streams, parse out XML Fragments, turn them
into packets, match them against classes, instantiate the right object,
serialize the xml into the object, and then raise the correct event.

Our SDK has source code available, and you can pull it down and look at how
we're doing everything:http://developers.coversant.net/Downloads/tabid/53/Default.aspx

What you're looking for is (mostly) found in the two classes:
Coversant.SoapBox.Base.XMPPSocket
Coversant.SoapBox.Base.XMLStreamReceiver

Our approach is very scalable, and quite reliable. Our use of socket buffers
that comes out of a buffer pool for reads/write, minimizes heap
fragmentation.

Thank you. I'll take a look at that over the weekend and see what I
can learn from it.
Thanks again for the time.