TcpClient lost data timing anomaly

[David J. Marcus]

Hi

The platform: XP/pro on a dual Xeon 3GH processors

I am using TcpClient class to communicate with a POP3 server.

Short problem synopsis: a 'read' hangs even though there is data in the
buffer (as evidenced by a network sniffer trace).

Elaboration:

I/O is done by getting the NetworkStream and doing a read for a whole
buffer's worth (typically 1024) of bytes.

The code looks for the NL (new-line char, '\n'), as a delimiter. If found,
the text up to and including the NL is returned to the caller. Otherwise
another read is performed and concatenated to the previous read's output,
and the check for the delimiter repeats. On subsequent reads, if there was
data left over from the previous read, it is processed for NL before any
other reads are issued.

I have used a network sniffer to determine that all the data is coming in.

I have put a trace in my program to output to a file (serialized with a
'lock') all the output form all the read requests.

I consistently find that one of the reads is able to get all but the last 3
bytes of a response from a server. The code dutifully returns all the text
received and tries to do another read since the caller is looking for a
'.\r\n' sequence (end-of-message). This read hangs (I use the debugger).

When I change the buffer size the problem happens in a different message. If
I make the buffer large enough, the problem goes away.

The buffering code described above is working for thousands of messages that
have split across multiple buffers or for short messages where multiple fit
in the same buffer, so I'm pretty confident that it is splitting messages
correctly and gluing together segments across network messages.

My only conclusion is that something in the TcpClient (or deeper) is
checking the socket and waiting for more data in a non-atomic manner. The
problem shows up on my system because it is a fast dual processor system
that can obviously do concurrent processing by multiple threads/processes.

Has any one seen this kind of problem before?

Any work-around?

-TIA
David

 

[David J. Marcus]

Recap of problem: Communications with a remote POP3 server hangs on a socket
read even though a network sniffer trace shows that the packet sent by the
server contains the needed characters. The socket read behaves as if no data
has arrived.

The platform: Win/XP pro, dual-processor 3Ghz Xeon processors with plenty of
RAM (1GB)

An update to the problem:

I eliminated the TcpClient class from the implementation. The code now uses
a socket to directly read/write from/to the POP3 server. The problem is
still there.

In particular, the problem is very sensitive to buffer size for socket
reads. When the buffer size is 1024, the communication with the server will
hang while retrieving some POP3 server responses (email messages). Other
buffersize values such as 1025 or other non-multiples of 1024 have no
problem.

The problem is entirely reproducible... that is, the hang happens only on
some messages and always on those messages.

As noted earlier, the network sniffer trace shows that the data was
correctly and completely received by the local machine (the sniffer shows
the correct contents in the packet).

I'm using code to always record (I/O trace) to a file the results of the
socket read. The I/O trace shows that the data is simply not returned by the
socket. By the way, the data that is 'lost' is 3 characters: a period, CR,
and NL (ie, ".\r\n" which is the end-of-message string used by the POP3
server to indicate no more data for the current request).

Interestingly enough, the problem seems to occur when the receipt of the
final 3 bytes would overflow the buffer.

Any ideas of how to fix this problem would be appreciated. And as I said,
the problem is reproducible

-TIA
David

btw, here's the basic socket 'read' I/O code: Note that every data read in
from the socket is logged to an external file.

in the class I have the following definitions
protected const int BUFFSIZE = 1024
protected byte[] m_ReadBuffer = new byte[BUFFSIZE];
protected string m_ReadCache = "";
protected Socket m_Socket;

//---------------------------------------------------------->
//-- get the next line segment from the server
public string ReadLine()
{
string line = "";
while (true)
{
//-- read more data if our local cache is empty
if (m_ReadCache.Length == 0)
{
int readLen = m_Socket.Receive(m_ReadBuffer, 0,
BUFFSIZE, SocketFlags.None);
if (readLen == 0)
break;

//-- use ASCII encoder to convert bytes to a string
m_ReadCache = m_Encoder.GetString(m_ReadBuffer, 0,
readLen);

//-- log message segment(s) to a log file
Logger.Log("SocketIO.ReadLine():",
"ReadCache=#" + m_ReadCache.Length + "='" +
m_ReadCache + "'");
}

//-- look for delimiter
int index = m_ReadCache.IndexOf('\n');
if (index >= 0)
{
line += m_ReadCache.Substring(0, index + 1);
m_ReadCache = m_ReadCache.Substring(index + 1);
break;
}

//-- no delimiter.. just accumaulate result
line += m_ReadCache;
m_ReadCache = "";
}

//-- the accumulated line up to and including the '\n'
return line;
}

 

[Yan-Hong Huang[MSFT]]

Hello David,

Thanks for posting in the group and the detailed problem description.

I haven't seen such problems before. After reviewing the whole thread, I
think we need to clarify some symptoms before digging into it.

1) Does this problem happen on other single CPU machines? Could you please
test it and post result here?

2) As you mentioned, the error only happens for some messages? Could you
please post a message sample here?

3) If using Win32 Socket API to repro it, could this problem be reproed?

Without digging into the debugging, we can't tell what happened in the
source. So the above informaiton is quite useful in troubleshooting. By the
way, I also notice from your description that chaning buffer size could
avoid this problem. For now, we could use it as a workaround temporarily.

Please post here if you have any more concerns on it.

Best regards,
Yanhong Huang
Microsoft Online Partner Support

Get Secure! - www.microsoft.com/security
This posting is provided "AS IS" with no warranties, and confers no rights.

--------------------
!From: "David J. Marcus" <[email protected]>
!References: <[email protected]>
!Subject: Re: TcpClient anomaly seems to be a socket problem
!Date: Sat, 13 Sep 2003 17:21:49 -0400
!Lines: 98
!X-Priority: 3
!X-MSMail-Priority: Normal
!X-Newsreader: Microsoft Outlook Express 6.00.2800.1158
!X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1165
!Message-ID: <[email protected]>
!Newsgroups: microsoft.public.dotnet.framework
!NNTP-Posting-Host: pcp01584522pcs.nrockv01.md.comcast.net 68.48.151.219
!Path: cpmsftngxa06.phx.gbl!TK2MSFTNGP08.phx.gbl!TK2MSFTNGP10.phx.gbl
!Xref: cpmsftngxa06.phx.gbl microsoft.public.dotnet.framework:53651
!X-Tomcat-NG: microsoft.public.dotnet.framework
!
!Recap of problem: Communications with a remote POP3 server hangs on a
socket
!read even though a network sniffer trace shows that the packet sent by the
!server contains the needed characters. The socket read behaves as if no
data
!has arrived.
!
!The platform: Win/XP pro, dual-processor 3Ghz Xeon processors with plenty
of
!RAM (1GB)
!
!An update to the problem:
!
!I eliminated the TcpClient class from the implementation. The code now uses
!a socket to directly read/write from/to the POP3 server. The problem is
!still there.
!
!In particular, the problem is very sensitive to buffer size for socket
!reads. When the buffer size is 1024, the communication with the server will
!hang while retrieving some POP3 server responses (email messages). Other
!buffersize values such as 1025 or other non-multiples of 1024 have no
!problem.
!
!The problem is entirely reproducible... that is, the hang happens only on
!some messages and always on those messages.
!
!As noted earlier, the network sniffer trace shows that the data was
!correctly and completely received by the local machine (the sniffer shows
!the correct contents in the packet).
!
!I'm using code to always record (I/O trace) to a file the results of the
!socket read. The I/O trace shows that the data is simply not returned by
the
!socket. By the way, the data that is 'lost' is 3 characters: a period, CR,
!and NL (ie, ".\r\n" which is the end-of-message string used by the POP3
!server to indicate no more data for the current request).
!
!Interestingly enough, the problem seems to occur when the receipt of the
!final 3 bytes would overflow the buffer.
!
!Any ideas of how to fix this problem would be appreciated. And as I said,
!the problem is reproducible
!
!-TIA
!David
!
!btw, here's the basic socket 'read' I/O code: Note that every data read in
!from the socket is logged to an external file.
!
!in the class I have the following definitions
! protected const int BUFFSIZE = 1024
! protected byte[] m_ReadBuffer = new byte[BUFFSIZE];
! protected string m_ReadCache = "";
! protected Socket m_Socket;
!
! //---------------------------------------------------------->
! //-- get the next line segment from the server
! public string ReadLine()
! {
! string line = "";
! while (true)
! {
! //-- read more data if our local cache is empty
! if (m_ReadCache.Length == 0)
! {
! int readLen = m_Socket.Receive(m_ReadBuffer, 0,
!BUFFSIZE, SocketFlags.None);
! if (readLen == 0)
! break;
!
! //-- use ASCII encoder to convert bytes to a string
! m_ReadCache = m_Encoder.GetString(m_ReadBuffer, 0,
!readLen);
!
! //-- log message segment(s) to a log file
! Logger.Log("SocketIO.ReadLine():",
! "ReadCache=#" + m_ReadCache.Length + "='" +
!m_ReadCache + "'");
! }
!
! //-- look for delimiter
! int index = m_ReadCache.IndexOf('\n');
! if (index >= 0)
! {
! line += m_ReadCache.Substring(0, index + 1);
! m_ReadCache = m_ReadCache.Substring(index + 1);
! break;
! }
!
! //-- no delimiter.. just accumaulate result
! line += m_ReadCache;
! m_ReadCache = "";
! }
!
! //-- the accumulated line up to and including the '\n'
! return line;
! }
!
!
!
!
!
!