C
Chris Mullins
We've been using the SSLStream class found in System.Net.Security to build a
giant Sockets server that provides TLS encryption at the channel leve.
Before .Net 2.0, we used an open-source encryption channel from Mentalis,
and have even looked at the Mono implementation for doing this.
The problem comes from the SSLStream not doing any buffer management. None.
Zero. In the "no buffer management" case, each SSLStream allocates bufferes
whenevers it needs them, copies data into those buffers, and then calls
Socket.BeingRead / Socket.BeginWrite. This causes pinning. Massive,
long-lived, horrible pins. This leads to horrendus non-recoverable
fragmentation.
The killer here is that both of these socket methods immediatly pin the
buffers, and pass the data off to unmanaged code. The BeginRead method may
not return for 10 seconds, 10 minutes, or 10 hours, leaving a hole in the
Managed Heap the entire time. BeginWrite causes the same problem, although
for a smaller length of time.
To get around this in the past, we've created a block of buffers ahead of
time, and cycled through them. This keeps all the pinned memory together in
a single spot. This technique is described in detail here:
http://blogs.msdn.com/yunjin/archive/2004/01/27/63642.aspx
http://www.coversant.net/dotnetnuke/Default.aspx?tabid=88&EntryID=9
In our last implementation (.Net 1.1), we put alot of work into managing
these buffer pools and really trying hard to eliminate heap fragmentation.
This worked great.
With the .Net 2.0 SSLStream, the BeginRead method ends up allocating
bufferes here:
BeginRead->ProcessRead->EnsureInternalBufferSize
That method allocates a buffer by:
this._InternalBuffer = new byte[addSize + curOffset];
Notice there's no fregging pool here? No attempt to eliminate fragmentation.
No attempt to be efficient.
This buffer gets passed to the NetworkStream, and in turn to the Socket,
which promptly pins it.
The BeginWrite case is the same thing:
BeginWrite->ProcessWrite->StartWriting->EncryptBuffers->EncryptData->Encrypt
This method allocates it's buffer:
buffer1 = new byte[(size + this.m_HeaderSize) + this.m_TrailerSize];
Again, no fregging attempt to use a buffer pool. No attempt to eliminate
heap fragmentation.
Because the SSLStream is retarded, it only works on NetworkStreams - and
Network Streams only work on Sockets. This means there's nowhere in the
whole chain I can insert code prior to the Pin that would use a pooled
buffer. Ugh.
Now, .Net 2.0 is improved at managing pinning in the heap:
http://blogs.msdn.com/maoni/archive/2005/10/03/so-what-s-new-in-the-clr-2-0-gc.aspx
.... but I have it on VERY good authority that their technique isn't nearly
as effective as the BufferPool method. In fact, it's not even close.
This has runied my whole day. Ugh.
giant Sockets server that provides TLS encryption at the channel leve.
Before .Net 2.0, we used an open-source encryption channel from Mentalis,
and have even looked at the Mono implementation for doing this.
The problem comes from the SSLStream not doing any buffer management. None.
Zero. In the "no buffer management" case, each SSLStream allocates bufferes
whenevers it needs them, copies data into those buffers, and then calls
Socket.BeingRead / Socket.BeginWrite. This causes pinning. Massive,
long-lived, horrible pins. This leads to horrendus non-recoverable
fragmentation.
The killer here is that both of these socket methods immediatly pin the
buffers, and pass the data off to unmanaged code. The BeginRead method may
not return for 10 seconds, 10 minutes, or 10 hours, leaving a hole in the
Managed Heap the entire time. BeginWrite causes the same problem, although
for a smaller length of time.
To get around this in the past, we've created a block of buffers ahead of
time, and cycled through them. This keeps all the pinned memory together in
a single spot. This technique is described in detail here:
http://blogs.msdn.com/yunjin/archive/2004/01/27/63642.aspx
http://www.coversant.net/dotnetnuke/Default.aspx?tabid=88&EntryID=9
In our last implementation (.Net 1.1), we put alot of work into managing
these buffer pools and really trying hard to eliminate heap fragmentation.
This worked great.
With the .Net 2.0 SSLStream, the BeginRead method ends up allocating
bufferes here:
BeginRead->ProcessRead->EnsureInternalBufferSize
That method allocates a buffer by:
this._InternalBuffer = new byte[addSize + curOffset];
Notice there's no fregging pool here? No attempt to eliminate fragmentation.
No attempt to be efficient.
This buffer gets passed to the NetworkStream, and in turn to the Socket,
which promptly pins it.
The BeginWrite case is the same thing:
BeginWrite->ProcessWrite->StartWriting->EncryptBuffers->EncryptData->Encrypt
This method allocates it's buffer:
buffer1 = new byte[(size + this.m_HeaderSize) + this.m_TrailerSize];
Again, no fregging attempt to use a buffer pool. No attempt to eliminate
heap fragmentation.
Because the SSLStream is retarded, it only works on NetworkStreams - and
Network Streams only work on Sockets. This means there's nowhere in the
whole chain I can insert code prior to the Pin that would use a pooled
buffer. Ugh.
Now, .Net 2.0 is improved at managing pinning in the heap:
http://blogs.msdn.com/maoni/archive/2005/10/03/so-what-s-new-in-the-clr-2-0-gc.aspx
.... but I have it on VERY good authority that their technique isn't nearly
as effective as the BufferPool method. In fact, it's not even close.
This has runied my whole day. Ugh.