Async Socket Disconnect Problem

[dougmcmurtry]

I have an Asynchronous socket that sends data to a server for credit
card approvals. The socket is kept alive by the server through a
heartbeat that sends a "beat" every 90 seconds. Trouble is that the
network is unreliable at times and thus the server will drop my
connection from time to time. I need to code around this so I can
reconnect to the server whenever this happens.

THE PROBLEM:
The Connected, Poll, and Available properties and methods are all
telling me that I am working, but I know for a fact that the server is
expecting me to reconnect because I stopped my heartbeat. This is all
verified so please don't question the scenario, I just need help
finding out that my connection is no longer valid through some
property on the .NET Socket or something in the Async callbacks that I
may be missing. Any help, please!!!!


Doug M.

 

[Peter Duniho]

I have an Asynchronous socket that sends data to a server for credit
card approvals. The socket is kept alive by the server through a
heartbeat that sends a "beat" every 90 seconds. Trouble is that the
network is unreliable at times and thus the server will drop my
connection from time to time. I need to code around this so I can
reconnect to the server whenever this happens.

THE PROBLEM:
The Connected, Poll, and Available properties and methods are all
telling me that I am working, but I know for a fact that the server is
expecting me to reconnect because I stopped my heartbeat. This is all
verified so please don't question the scenario, I just need help
finding out that my connection is no longer valid through some
property on the .NET Socket or something in the Async callbacks that I
may be missing. Any help, please!!!!

Well, the first advice: don't tell people what they may or may not
question in your post. You're asking a question in a public forum, and
it's rude to ask at the outset that respondents comply with your arbitrary
restrictions. If they feel it might be useful to question the scenario,
they should do so, and shouldn't feel limited by your own preferences.

Second advice: you should be clear about what network protocol you're
using. I'm going to assume TCP/IP because your problem description is
consistent with that, and that's usually what people are using when they
don't mention what they are using. But people reading your question
shouldn't have to guess.

Now, that said: the basic problem is that the properties you're looking at
only provide information that's as recent as the last time you tried to
send any data, because of the way that TCP/IP works. No error in the
connection is detected until you actually try to send data, and so as long
as the last time you tried to send data everything was fine, those
properties will indicate a valid, usable socket. Even if the server has
on its end closed the socket.

Now comes the point where I do question the scenario: you may not have
control over the server, but it's very bad design to have something like a
90 second "heartbeat" unless there's a very good reason to want to force
the connection to fail on 90 second intervals. The only reason you're
running into a problem at all is because of this "heartbeat"...even with
an unreliable network, your connection would remain intact assuming no
other attempts to communicate during moments of unreliability.

So, the bottom line here is that if you don't send anything to the server,
you will never detect an error via the built-in mechanisms. So either you
need to send something to the server so that the error does get detected,
or include your own timeout logic so that if you don't receive data from
the server within a 90 second time period (or maybe a little longer, just
to account for small delays that might happen when the server sends the
"heartbeat"), you know to close the socket and reopen the connection.

Pete

 

[Jeroen Mostert]

I have an Asynchronous socket that sends data to a server for credit
card approvals. The socket is kept alive by the server through a
heartbeat that sends a "beat" every 90 seconds. Trouble is that the
network is unreliable at times and thus the server will drop my
connection from time to time. I need to code around this so I can
reconnect to the server whenever this happens.

THE PROBLEM:
The Connected, Poll, and Available properties and methods are all
telling me that I am working, but I know for a fact that the server is
expecting me to reconnect because I stopped my heartbeat. This is all
verified so please don't question the scenario, I just need help
finding out that my connection is no longer valid through some
property on the .NET Socket or something in the Async callbacks that I
may be missing. Any help, please!!!!

There is no such property. This is an inherent feature/limitation of TCP/IP.
None of these properties work unless the connection is being actively used
(including active disconnection).

The *only* way to detect a broken connection is to send data on it and find
out that it doesn't work. You will only get an error on receiving data if
the other side shuts down the connection and your side sees it (which is not
the case for unexpected drops). In other words: you (the client) should be
sending heartbeats, and as soon as that fails, your connection will break
and you should reconnect. Ideally, the server should send replies to that
heartbeat that you can timeout on, so you can detect a one-sided connection.
The other way around works as well, but the bottom line is that you should
be sending.

If you *are* doing all that and you're still not detecting a broken
connection, you should check your firewall configuration (on either end). It
may be keeping the connection open improperly.

Note that even with heartbeats, you get no guarantee of uninterrupted
service, and sending a request can always fail, even if you just had a good
heartbeat. Your protocol should work exactly as well without using *any*
heartbeats and rebuilding the connection on the first send failure (since a
TCP/IP connection can remain open indefinitely through failures, as long as
it's up when it's actually used). Your protocol should have reasonable
timeouts for detecting these failures in an acceptable timeframe. The only
advantage to heartbeats is that you stand a better chance (on average) of
having a usable connection when a request comes in.

 

[dougmcmurtry]

Pete,

Sorry if offended anyone, I was just trying to impress that I know
that my test for getting the connection to drop is fine and I didn't
want people to get hung up on that detail. As for the heartbeat, this
is an old system that has thousands of POS systems requesting credit
card auths and the heartbeat is to let the server know that things are
still alive. Unfortunately I cannot go into any more detail, non-
disclosure. I thank you for your help and the scenario you described
is my current way of prevention, trouble is that if a request comes
through inbetween heartbeats I'll have no idea and the request will be
lost. Even bigger than that is that the BeginSend method and the
AsyncResult have no errors, exceptions, or invalid return data. So
there is my problem, no way of telling if the current send is getting
through. Thanks...


Doug M.

 

[Jeroen Mostert]

Even bigger than that is that the BeginSend method and the AsyncResult
have no errors, exceptions, or invalid return data.

They most assuredly do! In case of detected failure, your call to .EndSend()
will get the same SocketException as a synchronous call to .Send() would
get. You're either not seeing the connection fail (in which case, do check
if your network is perhaps misconfigured and making it impossible to detect
failure at all), or you're not calling .EndSend(). Please tell me it's not
the latter.

So there is my problem, no way of telling if the current send is getting
through. Thanks...

If the system doesn't have a reasonable timeout time in which the server is
supposed to respond to requests/heartbeats (or indeed if the server doesn't
respond at all!), I'd say that's a failure of the system to take the basic
nature of the network into account. Ditto if it has no way of safely
retrying a request without the server treating it as a duplicate.

I reiterate that a heartbeat does not solve connection problems. In the best
case it allows you to detect them sooner, but even that can be too much to
hope for. In the end, nothing beats a reliable network connection if you
need reliable service...

A kludgy, last-resortish but possibly helpful suggestion is to mess around
with the send buffer size (in particular, setting the send buffer to 0)
and/or disabling Nagle. This is generally *very unadvisable* as it can cause
severe throughput problems/network storm, but if detecting connecting
failure is more important than anything it's worth a shot.

 

[Peter Duniho]

Pete,

Sorry if offended anyone, I was just trying to impress that I know
that my test for getting the connection to drop is fine and I didn't
want people to get hung up on that detail. As for the heartbeat, this
is an old system that has thousands of POS systems requesting credit
card auths and the heartbeat is to let the server know that things are
still alive.

I realize that you may not control this, but regardless: it's a poor
design. You need to understand that you're trying to work with a system
that's poorly designed, and that this may create headaches for you that
you otherwise wouldn't have.

Unfortunately I cannot go into any more detail, non-
disclosure. I thank you for your help and the scenario you described
is my current way of prevention,

It may not matter, but which scenario? I described two approaches: send
data and detect the error at that point; or, maintain your own timeout
logic and detect an error based on the lack of the heartbeat after a
certain amount of time.

trouble is that if a request comes
through inbetween heartbeats I'll have no idea and the request will be
lost.

What request? Can a request come through other than between two
heartbeats? If not, how is the heartbeat behavior relevant?

It is a fundamental problem in communications (and not just those using
computers) that the recipient of a communication cannot know that they
have failed to receive a communication, unless some prearrangement has
been made so that they know to expect one. So, the heartbeat is an
example of a communication you can reliably know whether you got or not.
Any other communication is an example of a communication that is
_fundamentally impossible_ to know for sure that you failed to receive.

If you're trying to know that the server sent you something other than the
heatbeat and that you failed to receive that something, you seem to be
trying to solve an unsolveable problem.

Even bigger than that is that the BeginSend method and the
AsyncResult have no errors, exceptions, or invalid return data. So
there is my problem, no way of telling if the current send is getting
through. Thanks...

If you attempt to send data and it fails to be sent, you should receive an
error. BeginSend() won't return the error...instead, you'll get the error
when calling EndSend() in your callback (or wherever you do so, if you're
using some other mechanism to complete the send).

If you have a specific example of a situation in which you are sending
data that fails to be sent without you knowing about it, then you should
post a concise-but-complete sample of code that reliably demonstrates that
situation.

Note that in the case of network code, "complete" means both ends of the
connection. A code sample that doesn't provide 100% of the code required
to run and reproduce the problem is incomplete. A common complaint from
questioners is that they aren't in control of the other end, nor even have
code for that end (or are not allowed to share it). That's a true
complaint, but it doesn't change the fact that a complete sample is
required in order for someone else to understand the question and answer
it.

Pete