NetworkStream EndRead SocketException OperationAborted

[Guest]

I have a .NET 2.0 application that acts as a TCP/IP server, and I’ve
discovered an issue with NetworkStream. I’ve accepted a client socket and
called steam.BeginRead(…) and my AsyncCallback has been called, so it calls
EndRead. EndRead occasionally throws an IoException that has an
InnerException of SocketException that has a SocketErrorCode of
OperationAborted. The MSDN documentation says that OperationAborted means
the socket is closed, and the socket’s IsConnected does indeed indicate false.

But I handle the exception and call stream.BeginRead(..) again and it works
fine and I continue to have a conversation with my client like nothing ever
happened. The client sends me a message every 15 seconds, and I reply OK.
Additionally, I have a 1 second timer and I can continue to send unsolicited
messages to the client every second.

Running in the debugger, I notice that I always get an Output Window message
of “The thread 0x1234 has exited with code 0 (0x0).†just before I get the
exception with the OperationAborted. I’m not terminating any threads in my
code.

Why does EndRead throw that exception that indicates OperationAborted?
How should I handle this exception? I could close the socket, but that
basically means that my application isn’t servicing clients.
Why is the NetworkStream and the Socket still connected and working?

Thanks,
John

 

[Jeffrey Tan[MSFT]]

Hi John,

I paste the NetworkStream.EndRead code below. It will throw IOException in
various places. Since your IOException has the inner exception object, we
can determine that the exception is generated in the
streamSocket.EndReceive() method.

public override int EndRead(IAsyncResult asyncResult)
{
int num2;
if (this.m_CleanedUp)
{
throw new ObjectDisposedException(base.GetType().FullName);
}
if (asyncResult == null)
{
throw new ArgumentNullException("asyncResult");
}
Socket streamSocket = this.m_StreamSocket;
if (streamSocket == null)
{
throw new IOException(SR.GetString("net_io_readfailure", new
object[] { SR.GetString("net_io_connectionclosed") }));
}
try
{
num2 = streamSocket.EndReceive(asyncResult);
}
catch (Exception exception)
{
if (((exception is ThreadAbortException) || (exception is
StackOverflowException)) || (exception is OutOfMemoryException))
{
throw;
}
throw new IOException(SR.GetString("net_io_readfailure", new
object[] { exception.Message }), exception);
}
catch
{
throw new IOException(SR.GetString("net_io_readfailure", new
object[] { string.Empty }), new
Exception(SR.GetString("net_nonClsCompliantException")));
}
return num2;
}

EndReceive() method just wraps the errorCode returned and throws it out
with SocketException:

public int EndReceive(IAsyncResult asyncResult)
{
SocketError errorCode;
int num = this.EndReceive(asyncResult, out errorCode);
if (errorCode != SocketError.Success)
{
throw new SocketException(errorCode);
}
return num;
}

Regarding OperationAborted, I assume you mean WSA_OPERATION_ABORTED(995)
error code:
http://msdn2.microsoft.com/en-us/library/ms740668.aspx

I assume some code has canceled the overlapped operation, which caused the
WSA_OPERATION_ABORTED code generated from OS.

Is it possible for you to provide a sample project to help me reproduce
this problem locally? Then I can perform some debugging on it to find out
the root cause. Thanks.

Regarding the thread exiting in your application, it may be a thread pool
thread maintained by .Net for internal work. So it may exit without your
notice after completing certain work. Anyway, you may break your
application under debugger and examine all the threads call stacks to
understand each thread purpose. Then, when that thread exited silently, you
may look back in the previous threads list(using thread id) to see what is
the exited thread.

Anyway, if you can provide a sample project, I can help to find the reason
better. Thanks.

Best regards,
Jeffrey Tan
Microsoft Online Community Support
==================================================
Get notification to my posts through email? Please refer to
http://msdn.microsoft.com/subscriptions/managednewsgroups/default.aspx#notif
ications.

Note: The MSDN Managed Newsgroup support offering is for non-urgent issues
where an initial response from the community or a Microsoft Support
Engineer within 1 business day is acceptable. Please note that each follow
up response may take approximately 2 business days as the support
professional working with you may need further investigation to reach the
most efficient resolution. The offering is not appropriate for situations
that require urgent, real-time or phone-based interactions or complex
project analysis and dump analysis issues. Issues of this nature are best
handled working with a dedicated Microsoft Support Engineer by contacting
Microsoft Customer Support Services (CSS) at
http://msdn.microsoft.com/subscriptions/support/default.aspx.
==================================================
This posting is provided "AS IS" with no warranties, and confers no rights.

 

[Guest]

Jeffrey,
Thanks for the response.

I am using .NET 2.0 and can get the error code directly from the
SocketException.SocketErrorCode property which I assume matches the old WSA
error code. With .NET 2.0, there is no longer any reason to look at the old
WSA codes directly. I will say that I liked sockets in native C++ much more
that in C#, but that is a whole other post.

I have test code that does NOT show the problem. My real code shows the
problem only sometimes. I don’t think I can provide a reasonable volume of
code that shows the problem.

My TCP/IP based protocol uses a 12 byte header for both requests and
responses. That header contains an ID and the total length of the message;
some messages consist only of the header. My code uses BeginRead and EndRead
to get the 12 byte header, then uses a BinaryReader to get the rest of the
message (if there is more) and a BinaryWriter to send the response. This is
necessary since a Socket is not a stream in .NET, since there is no way to
read anything from a Socket except bytes, and since BinaryReader and
BinaryWriter do not support asynchronous operations. Using only Asynchronous
I/O would require buffering each entire message, which could be gigabytes.

Typically, a client connects and sends a 12 byte message which I respond to
with another 12 byte message. The client sends 12 more bytes and I respond
with 12 more bytes. After each response, I call BeginRead again. After this
second response, I either continue OK or get this SocketException for the
OperationAborted when I call EndRead. I'm testing using localhost and see
the problem about 1 of every 10 connections. If the problem does not happen
after the second message from the client, it either does not happen or does
not happen for several more minutes. It seems like a timing crack.

If I set a breakpoint before my first call to BeginRead, the problem never
happens; this makes it seem even more like a timing crack and also makes it
impossible for me to examine the threads before the problem happens. Since
the debugger message does not include a thread name, not even "", I am pretty
sure that it is an unmanaged thread that is exiting. The debugger typically
shows lots of threads that I did not create and if I try to see what these
threads are doing, the debugger only ever tells me that there is no source
code for that location.

I mentioned that the problem does not happen with my test code but does
happen with my real code. The real code is a large service that has much
more functionality than just being a TCP/IP server, so it has many .NET
threads that it creates itself and it also uses the thread pool. Both the
real code and the test code set the NetworkStream.ReadTimeout and
WriteTimeout. I do suspect that these timeouts are related to the problem,
and I suspect that the Socket is aborting because it is incorrectly timing
out the current BeginReceive when it intended to be checking the timeout on a
previous BeginReceive. Of course, this is just a hunch. I will continue my
attempts to make the problem happen in test code, but failing that will play
with the timeout values and see if the time it takes for the problem to
happen, when it does happen, correlates to the timeout value used.
John

 

[Jeffrey Tan[MSFT]]

Hi John,

Thank you for the detailed response.

Yes, based on your description, this problem seems like a timing related
issue. Actually, this type of issue is really difficult to troubleshoot.

Yes, the asynchronous I/O processing of socket code leverages the
ThreadPool threads for the asynchronized callback. So it is normal to see
the thread terminal under debugger.

Without a simple reproduce project, it is hard to me to help you find out
the root cause. The best way is obtaining a first chance full crash dump of
this Access Violation for analysis, however, this has gone out of the
newsgroup support boundary.

Looking at the nature of this issue, it would require intensive
troubleshooting which would be done quickly and effectively with direct
assistance from a Microsoft Customer Support Service. You can contact
Microsoft CSS directly to discuss additional support options you may have
available, at 1-(800)936-5800 or by choosing one of the options listed at
http://www.microsoft.com/services/microsoftservices/srv_support.mspx.

Anyway, if you are able to create a simple reproduce project, please feel
free to let me know.

Thank you for the understanding.

Best regards,
Jeffrey Tan
Microsoft Online Community Support
==================================================
Get notification to my posts through email? Please refer to
http://msdn.microsoft.com/subscriptions/managednewsgroups/default.aspx#notif
ications.

Note: The MSDN Managed Newsgroup support offering is for non-urgent issues
where an initial response from the community or a Microsoft Support
Engineer within 1 business day is acceptable. Please note that each follow
up response may take approximately 2 business days as the support
professional working with you may need further investigation to reach the
most efficient resolution. The offering is not appropriate for situations
that require urgent, real-time or phone-based interactions or complex
project analysis and dump analysis issues. Issues of this nature are best
handled working with a dedicated Microsoft Support Engineer by contacting
Microsoft Customer Support Services (CSS) at
http://msdn.microsoft.com/subscriptions/support/default.aspx.
==================================================
This posting is provided "AS IS" with no warranties, and confers no rights.