Socket Async Send I/O - Resource Leak / Callbacks not getting called?

[dennis.richardson]

Greetings all.

Here's a problem that's been driving me nuts for the last 48 hours.
I'm hoping that someone has come across this before.

I have a C# Application that reads a UDP broadcast (asynchronously).
Then it repackages these UDP packets and sends them to a subscriber
via TCP.

Now, I can read the UDP stream all day long without the application
ever using more memory. Everything get's GC'ed just fine.

If I connect a subscriber, and start dispatching the incoming packets
via BeginSend() everything goes according to plan.. for a short while.
Suddenly the application memory consumption starts increasing... and
I've profiled the memory with ".NET Memory Profiler".
It seems that there are many instances of something called
"OverlappedData" accumulating, which I think they mean to be:
"NativeOverlapped".
These are created, but never collected by the GC.

Yes.. I am calling "EndSend()" on the callback from "BeginSend()" in
case you are wondering.
But here's the thing... I've implemented a counter that increments
each BeginSend() dispatch and one counter that increments each
EndSend() callback. After a while I start seeing small differences
that do not go away after the GC kicks in, indicating that a few
Begins were naughty and didn't call back!!

I've googled everywhere and chased up any reference on resource leaks
in Async I/O under .NET but found nothing that indicated that
callbacks might get lost.

I'm at the end of my wits here. Hualp!! Has anyone ever encountered
something similar?

Many thanks in advance,

Dennis Richardson

PS: I've tried this under Windows Vista (64-bit) & Windows XP (32-
bit), no change whatsoever. I'm sure I'm doing something wrong... but
what?

 

[Guest]

Can you post a "short but complete" block of working sample code to
illustrate what you are doing that's causing the issue?
Peter

 

[Michael Rubinstein]

Hi Dennis,

I'm at the end of my wits here. Hualp!! Has anyone ever encountered
something similar?

yes, quite similar.

But here's the thing... I've implemented a counter that increments
each BeginSend() dispatch and one counter that increments each
EndSend() callback. After a while I start seeing small differences
that do not go away after the GC kicks in, indicating that a few
Begins were naughty and didn't call back!!

Do you experience data loss like packets sent, but never received by the
client?
Do you pass an 'out Error' object to BeginSend()? It would not change the
outcome, but could provide a clue.

My server applications get data over UDP, process it and pass processed
data to clients via TCP, same thing as you do. The memory use creeps up
until it reaches a certain limit, then GC 'wakes up' and reduces the amount
of memory for application use about 2.5 - 3 times and from that moment kept
at lower level then at program start. It may take a week before it happens.
I was quite nervous about this. My server applications run unattended for
several month. Since they never crash I am more confident by now. Still, I
have serious reservations about Asynchronous Sockets implementation under
..NET 2.0. A call back never called is just half of the potential problem.
The other half is when the call back is called but the result of Endxxx() is
a disposed object. I never experienced it with EndSend(), but with
EndReceive() it is 100% reproducible.

Michael

 

[Guest]

Can you post a "short but complete" block of working sample code to
illustrate what you are doing that's causing the issue?
Peter

Hello Peter,

Thanks for your response (and Michael too).
Not sure that a working example would be short, but here's basically what
I'm doing:

We have a main Async read loop:

public void OnReceive(IAsyncResult IAR)
{
// Read the data from the socket into the buffer.
// Then dispatch to TCP socket with BeginSend()
tcpSocket.BeginSend(... callback to OnSend);

// Queue next async read
udpSocket.BeginReceiveFrom(... callback to OnReceive);
}

public void OnSend(IAsyncResult)
{
// Do EndSend and be done...
}

Now, I have played around with it some more and discovered the following:

If the next async receive is delayed until the EndSend Callback was called,
then the problem doesn't appear.
(in other words OnReceive ->BeginSend->OnSend->BeginReceive->OnReceive... etc)
So somehow, if there is only one callback active at a time, then there is no
problem. I thought that it might have to do with ThreadPool threads
terminating and hence leaving the callbacks hanging (I saw a tidbit about
that somewhere).

Am I doing something wrong?

Anyway, does anyone have an idea of how to do a rather simple thing like
read UDP socket and push into one or more TCP sockets? Wouldn't one do it
sort of like I described?

I'm happy to provide more details, just don't want to flood the message with
lots of code.

Regards,

Dennis

 

[Guest]

Do you experience data loss like packets sent,
but never received by the client?

No, I'm not even worried about the client side at this stage. The server
crashing is enough of a problem.

The memory use creeps up until it reaches a certain limit,
then GC 'wakes up' and reduces the amount
of memory for application use about 2.5 - 3 times and from that moment kept
at lower level then at program start.

Well, these instances of "OverlappedData" (called so by the ".NET Memory
Profiler", I suspect it refers to isntances of NativeOverlapped objects) do
not get collected by the GC (no matter how many generations, full-depth GC
does nothing, so somehow there must be a reference to these things somewhere).

 

[Guest]

By the way, this guy had the same problem, apparently (didn't get an answer
though):

http://www.thescripts.com/forum/thread236349.html

 

[Willy Denoyette [MVP]]

Greetings all.

Here's a problem that's been driving me nuts for the last 48 hours.
I'm hoping that someone has come across this before.

I have a C# Application that reads a UDP broadcast (asynchronously).
Then it repackages these UDP packets and sends them to a subscriber
via TCP.

Now, I can read the UDP stream all day long without the application
ever using more memory. Everything get's GC'ed just fine.

If I connect a subscriber, and start dispatching the incoming packets
via BeginSend() everything goes according to plan.. for a short while.
Suddenly the application memory consumption starts increasing... and
I've profiled the memory with ".NET Memory Profiler".
It seems that there are many instances of something called
"OverlappedData" accumulating, which I think they mean to be:
"NativeOverlapped".
These are created, but never collected by the GC.

Yes.. I am calling "EndSend()" on the callback from "BeginSend()" in
case you are wondering.
But here's the thing... I've implemented a counter that increments
each BeginSend() dispatch and one counter that increments each
EndSend() callback. After a while I start seeing small differences
that do not go away after the GC kicks in, indicating that a few
Begins were naughty and didn't call back!!

That means that the send actually didn't occur , this means that all managed
and unmanaged resources associated with the logical connection (eg. Winsock
buffers) stay allocated which leads to an increased Private bytes
consumption. The GC cannot free this memory, nor should it because the
action did not complete yet. This would also mean that some subscribers
don't receive all of their expected data, right, now the questions are, does
it happen for particular , or arbitrary subscribers, how are they connected
(LAN, WAN, routed switched or both) how do you finally recover from this
(reconnect or restart the server?). Did you fire up a network trace?

Willy.

 

[Guest]

now the questions are, does
it happen for particular , or arbitrary subscribers, how are they connected
(LAN, WAN, routed switched or both) how do you finally recover from this
(reconnect or restart the server?). Did you fire up a network trace?

Thanks for your response Willy.

There's only one subscriber at the moment (haven't even added more than one)
where this happens. The subscriber seems to be fine, no problems on that end.
The two machines are on the same network segement (1GBit/s).

When you say the "sends didn't happen" wouldn't that imply that nothing
further should go through? In other words, the socket should be clogged?
Because that's not what I'm seeing. Yes, some callbacks don't seem to
complete, but the socket still transmits (the correct data).

It's a bit bizarre why this would happen, really, from a performance
standpoint as we are talking about only 1,000 UDP packets repacked into TCP
per second. So that can't be it?
To re-emphasize, this only happens on sending stuff. Receiving works fine
without any glitches.

Regards,

Dennis

 

[Michael Rubinstein]

Dennis,

No, I'm not even worried about the client side at this stage. The server
crashing is enough of a problem.

does it (crash)?

from your code:We have a main Async read loop:

public void OnReceive(IAsyncResult IAR)
{
// Read the data from the socket into the buffer.
// Then dispatch to TCP socket with BeginSend()
tcpSocket.BeginSend(... callback to OnSend);

// Queue next async read
udpSocket.BeginReceiveFrom(... callback to OnReceive);
}

public void OnSend(IAsyncResult)
{
// Do EndSend and be done...
}
I do very similar things. In my code I call udpSocket.BeginReceiveFrom(...
callback to OnReceive) before tcpSocket.BeginSend(... callback to OnSend).
Any particular reason for placing UDP async read after TCP BeginSend()? My
applications send processed UDP data to multiple TCP clients that connect
and disconnect 'at will'. I am uneasy about a few things in Async Socket
model, but it seems to be working.

Michael

 

[Guest]

I do very similar things. In my code I call udpSocket.BeginReceiveFrom(...
callback to OnReceive) before tcpSocket.BeginSend(... callback to OnSend).
Any particular reason for placing UDP async read after TCP BeginSend()?

Thanks Michael. The reason I did it in that order (BeginReceive after
BeginSend) is because I wanted to be reasonably sure that not another
"OnReceive" jumped in there and fired off a BeginSend before this one (which
could then be possible (although probably quite unlikely).
I will give this a try, although I have little hope that it'll make a
difference.

 

[Willy Denoyette [MVP]]

Thanks for your response Willy.

There's only one subscriber at the moment (haven't even added more than
one)
where this happens. The subscriber seems to be fine, no problems on that
end.
The two machines are on the same network segement (1GBit/s).

That means no routers, only a Gigabit switch switch? Also this happens on a
connected socket, here I mean a socket that stays connected for the duration
of the program and does not close/reconnect for each number of transfers.
What's the source of the UDP packets? is it the same "subscriber" machine,
or are they different?

When you say the "sends didn't happen" wouldn't that imply that nothing
further should go through? In other words, the socket should be clogged?
Because that's not what I'm seeing. Yes, some callbacks don't seem to
complete, but the socket still transmits (the correct data).

Well, I really meant that the send did not complete at the application
level, from your description it's clear that all data was correctly
transmitted, only thing is, the callback did not happen, this can be the
result of a failure to signal the IOC port (kernel space), but it can also
be a problem at a lower level in the kernel or on the network stack, for
instance with the NIC's driver code or the NDIS, IP, TCP or Winsock code.
Point is, your EndSend is never called so the underlying resources (managed
and unmanaged) are never freed (well, only when the process ends).
Don't know if you can test at a lower rate, say 100MB/sec and try again, if
the problem remains you can (possibly) exclude Gigabit related issues
(timing problems in the NIC and switches) and take a closer look at your
code, if the problem goes away, maybe you can try to replace the NIC (and
the switch) and find out with the vendor(s) whether this is not a known
problem.

It's a bit bizarre why this would happen, really, from a performance
standpoint as we are talking about only 1,000 UDP packets repacked into
TCP
per second. So that can't be it?

No, at least it shouldn't, don't know what would happen when you only send
data over TCP to the subscriber, say you simulate that repacking of UDP
messages into TCP streams and send these asynchronously.

I don't know if you could post a small but complete sample that illustrates
the issue, I have a similar Gigabit set-up which I could use to try it out
myself.

To re-emphasize, this only happens on sending stuff. Receiving works fine
without any glitches.

Willy.

 

[Guest]

What's the source of the UDP packets? is it the same "subscriber" machine,
or are they different?

Same subscriber machine. There's a process there that multicasts only on
this machine (can't do anything about this, out of my control). Don't think
that's the issue though?

I don't know if you could post a small but complete sample that illustrates
the issue, I have a similar Gigabit set-up which I could use to try it out
myself.

I will try to put something together. It's pretty clear to me that I'm doing
something wrong else I would encounter this "issue" more frequently on my web
searches. After all, applications that communicate via UDP and TCP are rather
common.

Do you see any issue dispatching the BeginSend from the OnReceive callback?

Regards,

Dennis

 

[Guest]

Okay, some more investigative results:

It seems that the problem might be that packets arrive too fast (at peak
load) sometimes for .NET to be able to handle repacking it into TCP. Baseload
might only be 1000 messages per second, but I think peak throughput might
exceed that by an order of magnitude.

I can simulate the same behavior by calling BeginSend a few times per
OnReceive. This just seems to swamp the framework with NativeOverlapped
instances that it doesn't collect.

The collection can be enforced by calling GC.Collect() many many times. Per
each GC(2nd generation) sweep, a few of them get removed. After a dozen or
so, manually triggered, rapid-fire GCs, they get collected. But that can be
hardly the point. It sees the CLR has problems freeing these for some reason.

I'm sort of stumped now what to do really? Maybe I should architect
differently?

 

[Michael Rubinstein]

Dennis, does your program actually run out of memory or you just assumed
it eventually will? Forcing GC.Collect() is never a good idea. In my
experience GC works best when the application code does not interfere
directly. I would be more concerned about packets sent but never received by
the TCP client. This would be a clear indication that something is wrong.Maybe I should architect differently?You could store incoming udp packets to a temporary buffer until it either
reaches a certain byte limit or a preset time interval expires, and then
BeginSend() the bundle at once. I am basically in 'the same boat'. I did not
take this approach jet because at the current work load my server seems to
manage. It would be more complex in my case. My udp packets are re-routed to
specific clients, one packet may be sent to one group of clients, the next
to a different group, and so on.

Michael

 

[Willy Denoyette [MVP]]

Okay, some more investigative results:

It seems that the problem might be that packets arrive too fast (at peak
load) sometimes for .NET to be able to handle repacking it into TCP.
Baseload
might only be 1000 messages per second, but I think peak throughput might
exceed that by an order of magnitude.

I could be wrong, but it looks like you are exhausting the IOC thread pool!
This pools has a max. of 1000 threads, did you ever inspect (using
ThreadPool.GetAvailableThreads ) the number of IO threads remaining in the
pool when receiving the UDP packets? Or simply did you ever inspect the
number of threads in the process?
I would love to see some code, really.

Willy.

 

[Chris Mullins [MVP]]

I would love to see some code, really.

I second that.

I'm suspicious of something strange going on in your code. With overlapped
structures sitting around not being collected, and sockets not responding
quickly enough, something weird is happening.