Possible bug in .net sockets implementation? Reproducible code inside

[jack]

At this link I have two c# projects, one is a client, the other is a
server. Just point the ip address of the client at the server

http://www.slip-angle.com/hosted/bug/


The server does little more than the examples in the c# documentation
for how to set up an asynchronous server. It just accepts connections,
reads data into a buffer continously, doing nothing with it.

The client connects 200 sockets, sends some data with each continously,
and occasionally disconnects a socket and reconnects a new one.

Things of note:

1. in this scenario memory increases forever. Not just the private
bytes but the gen2 heap increases forever, even after a forced gen2
garbage collection

2. memory profilers indicate no left behind object references

3.the memory profiler indicates "holes" in the gen2 heap, which can be
caused by "pinned objects". I think perhaps that the data in the
receive buffers is being held onto indefinitely, despite all sockets
being closed on the server.


4. If you change the client to not send any data, the memory will not
increase. If you change the client to not disconnect and reconnect any
sockets, memory will not increase.

If someone can verify if this is a .net bug, or point out my idiocy I
would greatly appreciate it.

 

[jack]

Some more info:

trying this test with 10 sockets will not lead to memory increase, but
with 50 sockets it will. odd!

 

[Willy Denoyette [MVP]]

At this link I have two c# projects, one is a client, the other is a
server. Just point the ip address of the client at the server

http://www.slip-angle.com/hosted/bug/


The server does little more than the examples in the c# documentation
for how to set up an asynchronous server. It just accepts connections,
reads data into a buffer continously, doing nothing with it.

The client connects 200 sockets, sends some data with each continously,
and occasionally disconnects a socket and reconnects a new one.

Things of note:

1. in this scenario memory increases forever. Not just the private
bytes but the gen2 heap increases forever, even after a forced gen2
garbage collection

2. memory profilers indicate no left behind object references

3.the memory profiler indicates "holes" in the gen2 heap, which can be
caused by "pinned objects". I think perhaps that the data in the
receive buffers is being held onto indefinitely, despite all sockets
being closed on the server.


4. If you change the client to not send any data, the memory will not
increase. If you change the client to not disconnect and reconnect any
sockets, memory will not increase.

If someone can verify if this is a .net bug, or point out my idiocy I
would greatly appreciate it.

No, this is not a bug in the socket library, this relates to the way you
handle your receive buffers and the way your testclient is implemented.
Let me try to explain what happens(I hope I succeed).

You noticed two important things:
1. The Gen2 heap keeps growing, while the number of objects don't.
2. There are holes in the Gen2 heap.

Now if you look at the #Pinned objects perf. counter, you will see there are
a considerable number of them.
These objects are long lived so they end in the Gen2 heap, worse, they end
near the top of the heap, as such preventing a compactation of the heap.
Note that collection is not prevented by this, which explains the ( nearly)
constant # of objects.

Now, why are objects aging into gen2, which objects are these and which are
the pinned objects?
These objects are (1)the MsgState instances, they get instantiated when the
Client connects and they live until the client closes the outgoing socket.
The MsgState contains a reference (2) dataBuffer to a byte[] used as socket
buffer, you know by now that these are pinned for the duration of the
BeginReceive/EndReceive and ProcessMessage sequence. But the problem (lets
call it 'issue1') is that their lifetime is tied to the lifetime of the
MsgState object.
Great, but why are they aging into gen2? This is related to the large number
of objects (sockets) and the way your client is implemented.
Your client creates 200 sockets, sends a (too) small buffer to each of them
in sequence, when done it closes and re-opens one single socket in the
sequence they were created. At that time the Socket (and related objects)
it's MsgState and dataBuffer become eligible for collection. Lets say the GC
kicks in, he won't have a problem to collect the garbage, but heck, he can't
compact the heap (Gen0 to start with) because there are pinned objects at
the top of it preventing reallocation, the result is a (small) hole. After
each GC run, the hole becomes bigger as the same pinned object (your lastly
created socket buffer) prevents compactation.
At some point in time, the Gen0 pointer gets adjusted and becomes the Gen1
pointer and after another number of GC runs the Gen1 pointer becomes the
Gen2 pointer, but at/near the top of it you will always find a (number of)
pinned object.

Ok, how to solve this?
1. In ProcessMessage, you should move the contents of the (pinned) receive
buffer to another buffer and release the buffer (re-create a new instance).
This will shorten the period the buffer is pinned, so it won't end as an
aged object.
2. You should change your test method (or at least your message sending
algorithm), I understand that you are trying to simulate a large number of
clients, but this is not the right way of doing. You need separate client
machines, and each one must run a number of processes and each process must
consist of a number of threads each accessing a single socket.

Another remark is that you are touching the UI from thread pool threads,
this is a definitive no no in windows, you have to marshal the call sing
Control.Invoke.

Willy.

 

[jack]

Thanks, I will try moving the contents of the buffer and recreating it
see if that does the trick (though I think I used to do it that way
with the same problem).

As for my test method, I wrote that test to simulate what is actually
happening on our live server, and it is causing the same result.

 

[jack]

Well, making copies of the msgstate and freeing up the old one seems to
have done the trick with my test case.

Very interesting. I obviously need to read up more on how the
compacting process works in .net

 

[Willy Denoyette [MVP]]

Thanks, I will try moving the contents of the buffer and recreating it
see if that does the trick (though I think I used to do it that way
with the same problem).

As for my test method, I wrote that test to simulate what is actually
happening on our live server, and it is causing the same result.

No, that alone won't do the trick, you also have to change the clients test
algorithm, make it asynchronous too, and use a number of multithreaded
clients handling a single or a limited number of sockets per thread.
Both issues are related, not releasing the buffers at EndReceiveData keep
them pinned for the duration of the connection, but that's only part of the
problem. Creating that large number of sockets in sequence followed by
sending data in the same sequence will result in a lot of aged/pinned
objects at the server.
Don't forget that the way you did, it takes 400 seconds before each socket
has received it's 200 messages and gets closed.

I'm also not clear on why you send such small buffers (only 6 bytes per
send), this is a very inefficient way of data transfer over TCP sockets

Willy.

 

[jack]

I'm simulating what will happen on our live server. Clients will
connect (a few hundred of them) occasionally send a small bit of data,
and disconnect after a while. These clients are out there in the
population so I have to deal with them somehow.

 

[Willy Denoyette [MVP]]

I'm simulating what will happen on our live server. Clients will
connect (a few hundred of them) occasionally send a small bit of data,
and disconnect after a while. These clients are out there in the
population so I have to deal with them somehow.

Jack,

I understand, but IMO it's not very realistic (not impossible) that a client
waits 2 seconds after a connect request before sending part of the data,
wait again 2 seconds and wait... etc. and finaly need 6 minutes to send 1200
bytes to a server. Wouldn't it be better to collect the data at the client
and send it in one go?

Just my 2c.
Willy.

 

[jack]

The clients allow realtime human interaction (chat, forwarding browser
events etc). If the user wants to send a short chat message every few
seconds, or send scroll events in realtime from time to time, that is
what they will do. And they might be in a session for an hour or more.

 

[jack]

Anyone else care to comment?

It seems kind of ridiculous that dot net can't support behavior like
this?

How would one go about writing a large scale server that deals with
realtime input from users that come and go?

 

[Willy Denoyette [MVP]]

Anyone else care to comment?

It seems kind of ridiculous that dot net can't support behavior like
this?

How would one go about writing a large scale server that deals with
realtime input from users that come and go?

What do you mean, I thought this was solved by copying your message state.

Willy.

 

[Cor Ligthert]

Jack,

You are using a solution from the beginning of Internet.

Ever thought of using a webservice. That makes you less dependend from all
kinds of hardware.

Just my thought,

Cor

 

[jack]

We have thought about it, thought I'm not sure how well that would work
with realtime interactions that have to go both directions (clients
pushing to server and server pushing to clients).

Willy, copying the message states made my test case stop growing
memory, but not our real production server.

I did just this morning come up with a solution that seems to have done
the trick.

I made a pool of message states. Grab one from the pool as sockets
connect, put them back as they disconnect.

 

[Willy Denoyette [MVP]]

We have thought about it, thought I'm not sure how well that would work
with realtime interactions that have to go both directions (clients
pushing to server and server pushing to clients).

Willy, copying the message states made my test case stop growing
memory, but not our real production server.

I did just this morning come up with a solution that seems to have done
the trick.

I made a pool of message states. Grab one from the pool as sockets
connect, put them back as they disconnect.

Great, I have written a similar test case as yours, the only thing I did is
make sure that the pinned IO buffers are decoupled from the state objects.
I'm actually running this code on a server (W2K3) with 2000 sockets
connected over a real network (not the loopback interface), managed heap
consumption seems stable until now.

Willy.

 

[jack]

If you are curious to see what I've been working on, you can check it
out here:

www.jybe.com

Thanks for your assistance!