Socket.BeginReceive does not handle incoming data immediately

[Bruce]

Hi
I have a problem with the BeginReceive method in Socket class while
using it to receive responses from a server. I have reread my code
many times but cannot figure out why the delay is happening.

My scenario is this:
1. I send a request for user1 to a server that will trigger 3
responses. (I call Send() method below)
2. I get the first response (I dont care about next 2)
3. I send a request for the next user (user2). (I call Send() method
again)
4. I get the pending 2 responses for the first request.
5. I get the first response for user2

1 through 4 happen almost at the same time. But between 4 and 5, there
is always ~200 millisecs delay. (The server responds to all requests
immediately)

Now, if I wait for all 3 responses at 2. then everything works fine
and I get all 3 responses within ~10ms (expected). But in the above
scenario, BeginReceive does not trigger the callback for 5. for
atleast 200 milliseconds.

Is there a problem with the way I am using the socket? I think it is
because I am doing a send when server is trying to send something and
because of that, I am having this issue.

~~
socket = new Socket(AddressFamily.InterNetwork, SocketType.Stream,
ProtocolType.Tcp);
socket.SetSocketOption(SocketOptionLevel.Socket,
SocketOptionName.Linger, false);

void Send(Socket socket, byte[] requestBytes) {
socket.BeginSend(requestBytes, 0, requestBytes.Length,
SocketFlags.None,
delegate(IAsyncResult
ar)
{

socket.EndSend(ar);
},
null);
}


void Receive(Socket socket) {
// 1 response contains 10 bytes
socket.BeginReceive(dataBuffer, 0, 10, SocketFlags.None, new
AsyncCallback(Callback), socket)
}

void Callback(IAsyncResult ar) {
/// process response
Receive(socket);
}

 

[Peter Duniho]

[...]
1 through 4 happen almost at the same time. But between 4 and 5, there
is always ~200 millisecs delay. (The server responds to all requests
immediately)

Does the server send the same three responses for the second user? Or is
the response different?

It's possible you're just seeing the normal behavior that the Nagle
algorithm imposes (coalesces network data to make transmission more
efficient, but it requires waiting briefly after short sends to see if any
more data can be included with it).

Now, if I wait for all 3 responses at 2. then everything works fine
and I get all 3 responses within ~10ms (expected).

Which 3 responses? Your question mentions four. If the server sends the
first three all at once, it's not surprising that if you actually try to
read them all at once, they are there.

But in the above
scenario, BeginReceive does not trigger the callback for 5. for
atleast 200 milliseconds.

Is there a problem with the way I am using the socket? I think it is
because I am doing a send when server is trying to send something and
because of that, I am having this issue.

The socket shouldn't have any trouble at all with simultaneous sends and
receives. But there are some ambiguities in your problem description that
make it difficult to say what you should expect and what might be causing
it.

I strongly suspect that the issue is not at your end, and that you are
getting notification of data received as soon as it gets to your
computer. Even if there was some delay due to the i/o thread waiting to
be run, it shouldn't be a 200ms delay (unless, I suppose, you've got a
single CPU and four threads or so that are constantly running and CPU
bound...but you didn't mention anything like that). So most like it just
means the data hasn't even show up until then.

You can get Wireshark, a free network analyzer, and watch the data coming
through and see exactly what happens when. That will give you a better
idea as to where to look.

That said, if a 200ms delay is not tolerable, you might want to reconsider
using TCP. It's just not intended for time-sensitive communications
(TCP/IP isn't as a whole, for that matter, though you can sometimes get
closer to your goal using UDP, depending on what that goal is).

Pete

 

[Bruce]

Hello Pete
Thanks for the reply.

Does the server send the same three responses for the second user?

No. The 3 responses themselves are different for a single user, but
for every user, the formats of these 3 responses are the same.
The actual content is different since they contain the user name and
other user specific information.

Which 3 responses? Your question mentions four.

I was referring to the 3 responses for the first request. (Send
request 1, do not send request 2, receive all 3 responses for request
1 -> this happens in 20 ms. Now send request 2, receive all 3
responses for request 2 in 20 ms)
I have 2 CPUs on the client - I will increase the size of the thread
pool just to make sure that is not an issue.

My trouble with this problem is - what is causing this huge time gap?
Since the server scales great and responds immediately (it has been
well tested and I looked at its logs etc), I must be doing something
incorrect in my client code - so I thought.
It looks like something is blocking for those 200 ms. I will use
Wireshark or netmon and capture a trace.

If you have more thoughts, please let me know.

Thanks again
Bruce

[...]
1 through 4 happen almost at the same time. But between 4 and 5, there
is always ~200 millisecs delay. (The server responds to all requests
immediately)

Does the server send the same three responses for the second user? Or is
the response different?

It's possible you're just seeing the normal behavior that the Nagle
algorithm imposes (coalesces network data to make transmission more
efficient, but it requires waiting briefly after short sends to see if any
more data can be included with it).

Now, if I wait for all 3 responses at 2. then everything works fine
and I get all 3 responses within ~10ms (expected).

Which 3 responses? Your question mentions four. If the server sends the
first three all at once, it's not surprising that if you actually try to
read them all at once, they are there.

But in the above
scenario, BeginReceive does not trigger the callback for 5. for
atleast 200 milliseconds.

Is there a problem with the way I am using the socket? I think it is
because I am doing a send when server is trying to send something and
because of that, I am having this issue.

The socket shouldn't have any trouble at all with simultaneous sends and
receives. But there are some ambiguities in your problem description that
make it difficult to say what you should expect and what might be causing
it.

I strongly suspect that the issue is not at your end, and that you are
getting notification of data received as soon as it gets to your
computer. Even if there was some delay due to the i/o thread waiting to
be run, it shouldn't be a 200ms delay (unless, I suppose, you've got a
single CPU and four threads or so that are constantly running and CPU
bound...but you didn't mention anything like that). So most like it just
means the data hasn't even show up until then.

You can get Wireshark, a free network analyzer, and watch the data coming
through and see exactly what happens when. That will give you a better
idea as to where to look.

That said, if a 200ms delay is not tolerable, you might want to reconsider
using TCP. It's just not intended for time-sensitive communications
(TCP/IP isn't as a whole, for that matter, though you can sometimes get
closer to your goal using UDP, depending on what that goal is).

Pete

 

[Bruce]

Hello Pete
Thanks for the reply.>Does the server send the same three responses for the second user?

No. The 3 responses themselves are different for a single user, but
for every user, the formats of these 3 responses are the same.
The actual content is different since they contain the user name and
other user specific information.

Which 3 responses?  Your question mentions four.

I was referring to the 3 responses for the first request. (Send
request 1, do not send request 2, receive all 3 responses for request
1 -> this happens in 20 ms. Now send request 2, receive all 3
responses for request 2 in 20 ms)
I have 2 CPUs on the client - I will increase the size of the thread
pool just to make sure that is not an issue.

My trouble with this problem is - what is causing this huge time gap?
Since the server scales great and responds immediately (it has been
well tested and I looked at its logs etc), I must be doing something
incorrect in my client code - so I thought.
It looks like something is blocking for those 200 ms. I will use
Wireshark or netmon and capture a trace.

If you have more thoughts, please let me know.

Thanks again
Bruce

[...]
1 through 4 happen almost at the same time. But between 4 and 5, there
is always ~200 millisecs delay. (The server responds to all requests
immediately)

Does the server send the same three responses for the second user?  Oris
the response different?

It's possible you're just seeing the normal behavior that the Nagle
algorithm imposes (coalesces network data to make transmission more
efficient, but it requires waiting briefly after short sends to see if any
more data can be included with it).

Which 3 responses?  Your question mentions four.  If the server sends the
first three all at once, it's not surprising that if you actually try to
read them all at once, they are there.

The socket shouldn't have any trouble at all with simultaneous sends and
receives.  But there are some ambiguities in your problem description that
make it difficult to say what you should expect and what might be causing
it.

I strongly suspect that the issue is not at your end, and that you are
getting notification of data received as soon as it gets to your
computer.  Even if there was some delay due to the i/o thread waiting to
be run, it shouldn't be a 200ms delay (unless, I suppose, you've got a
single CPU and four threads or so that are constantly running and CPU
bound...but you didn't mention anything like that).  So most like it just
means the data hasn't even show up until then.

You can get Wireshark, a free network analyzer, and watch the data coming
through and see exactly what happens when.  That will give you a better
idea as to where to look.

That said, if a 200ms delay is not tolerable, you might want to reconsider
using TCP.  It's just not intended for time-sensitive communications
(TCP/IP isn't as a whole, for that matter, though you can sometimes get
closer to your goal using UDP, depending on what that goal is).

Pete- Hide quoted text -

- Show quoted text -

To add on, I got the network traces using netmon, and they showed that
the message was actually got by my system 200 ms late. So, it looks
like my code does not introduce the delay.
Now, though the server sent it immediately (I should use netmon there
too to verify), I got it at my system 200 ms later...I cannot explain
it.
Is it possible that I am doing some socket operation on my client that
is blocking the server from sending me the packet immediately? (I am
not too conversant with how sockets exactly work.)

Thanks
Bruce

 

[Bruce]

Hello Pete
Thanks for the reply.>Does the server send the same three responses for the second user?

No. The 3 responses themselves are different for a single user, but
for every user, the formats of these 3 responses are the same.
The actual content is different since they contain the user name and
other user specific information.

Which 3 responses?  Your question mentions four.

I was referring to the 3 responses for the first request. (Send
request 1, do not send request 2, receive all 3 responses for request
1 -> this happens in 20 ms. Now send request 2, receive all 3
responses for request 2 in 20 ms)
I have 2 CPUs on the client - I will increase the size of the thread
pool just to make sure that is not an issue.

My trouble with this problem is - what is causing this huge time gap?
Since the server scales great and responds immediately (it has been
well tested and I looked at its logs etc), I must be doing something
incorrect in my client code - so I thought.
It looks like something is blocking for those 200 ms. I will use
Wireshark or netmon and capture a trace.

If you have more thoughts, please let me know.

Thanks again
Bruce

[...]
1 through 4 happen almost at the same time. But between 4 and 5, there
is always ~200 millisecs delay. (The server responds to all requests
immediately)

Does the server send the same three responses for the second user?  Oris
the response different?

It's possible you're just seeing the normal behavior that the Nagle
algorithm imposes (coalesces network data to make transmission more
efficient, but it requires waiting briefly after short sends to see if any
more data can be included with it).

Which 3 responses?  Your question mentions four.  If the server sends the
first three all at once, it's not surprising that if you actually try to
read them all at once, they are there.

The socket shouldn't have any trouble at all with simultaneous sends and
receives.  But there are some ambiguities in your problem description that
make it difficult to say what you should expect and what might be causing
it.

I strongly suspect that the issue is not at your end, and that you are
getting notification of data received as soon as it gets to your
computer.  Even if there was some delay due to the i/o thread waiting to
be run, it shouldn't be a 200ms delay (unless, I suppose, you've got a
single CPU and four threads or so that are constantly running and CPU
bound...but you didn't mention anything like that).  So most like it just
means the data hasn't even show up until then.

You can get Wireshark, a free network analyzer, and watch the data coming
through and see exactly what happens when.  That will give you a better
idea as to where to look.

That said, if a 200ms delay is not tolerable, you might want to reconsider
using TCP.  It's just not intended for time-sensitive communications
(TCP/IP isn't as a whole, for that matter, though you can sometimes get
closer to your goal using UDP, depending on what that goal is).

Pete- Hide quoted text -

- Show quoted text -

To add on, I got the network traces on client and server using netmon,
and I find something really weird.
The server is responding to all requests with the 3 responses
immediately. But the client gets the response only after 200 ms.
(netmon confirms this)
For example, a response sent by the server at time t is shown in my
client trace to have been got at t + 200 ms!
And the server trace shows that it got request 1, it sent back 3
responses, it got request 2 200 ms later, and it immediately sent back
3 responses.
(contradicting what the client trace says, which is: client sent
request, but got response 200 ms later)

So, is there something I am doing with my socket that is blocking the
message from being received by my system itself?
I am not very conversant with how sockets work, but this is really
confusing.

Thanks
Bruce

 

[Peter Duniho]

[...]
To add on, I got the network traces using netmon, and they showed that
the message was actually got by my system 200 ms late. So, it looks
like my code does not introduce the delay.

Right, as I suspected.

Now, though the server sent it immediately (I should use netmon there
too to verify), I got it at my system 200 ms later...I cannot explain
it.

You should use something to verify that, yes. I suspect that you'll find
that even though the server code has no delay within it, the data isn't
transmitted until later.

This would be the Nagle algorithm in operation. It's a fundamental part
of the design of the TCP protocol.

Is it possible that I am doing some socket operation on my client that
is blocking the server from sending me the packet immediately? (I am
not too conversant with how sockets exactly work.)

No, not really. If you were failing to read from the socket over a period
of time, that would cause the network driver buffer to fill up, preventing
more data from being sent to it. But that would not in and of itself
cause a delay. It could be cause _by_ some other delay, but it wouldn't
be the actual source of the delay.

If you are new to network code, you should probably read the Winsock FAQ
(http://tangentsoft.net/wskfaq/), and perhaps other network-related
resources.

I believe that the bottom line here is that what you're seeing is a basic
behavior of TCP, and short of disabling Nagle for a socket, there's not
much you can do about it (do that at your own risk...for low-volume stuff,
it's probably okay, but doing so can really mess up performance in a
high-volume application).

Pete

 

[Bruce]

[...]
To add on, I got the network traces using netmon, and they showed that
the message was actually got by my system 200 ms late. So, it looks
like my code does not introduce the delay.

Right, as I suspected.

Now, though the server sent it immediately (I should use netmon there
too to verify), I got it at my system 200 ms later...I cannot explain
it.

You should use something to verify that, yes.  I suspect that you'll find  
that even though the server code has no delay within it, the data isn't  
transmitted until later.

This would be the Nagle algorithm in operation.  It's a fundamental part 
of the design of the TCP protocol.

Is it possible that I am doing some socket operation on my client that
is blocking the server from sending me the packet immediately? (I am
not too conversant with how sockets exactly work.)

No, not really.  If you were failing to read from the socket over a period  
of time, that would cause the network driver buffer to fill up, preventing 
more data from being sent to it.  But that would not in and of itself  
cause a delay.  It could be cause _by_ some other delay, but it wouldn't 
be the actual source of the delay.

If you are new to network code, you should probably read the Winsock FAQ  
(http://tangentsoft.net/wskfaq/), and perhaps other network-related  
resources.

I believe that the bottom line here is that what you're seeing is a basic  
behavior of TCP, and short of disabling Nagle for a socket, there's not  
much you can do about it (do that at your own risk...for low-volume stuff, 
it's probably okay, but doing so can really mess up performance in a  
high-volume application).

Pete

Hi Pete
Thanks for your help. I guess it is just an optimization done by TCP
as you have explained. Now, I have learn something new - the Nagle
algorithm.
I have looked at all the traces again and it looks like till the
client sends back an ACK, the server does not send all the responses
(causing the 200 ms delay)

-Bruce