Interesting Client Server behavior - No connection could be madebecause the target machine actively

[rasta475]

I've got a handheld application (Windows Mobile 2003, CF 1.1) and a
server application (Windows Server 2000+, Framework 2.0) and the
purpose of these two pieces of software is to send collected data
(Orders) from the handheld to the server as well as send updated
reference data from the server to the handheld. This all works fine
in our test environment which consists of a Windows XP PC running
ActiveSync, a Server Class machine running Windows Server 2003, and a
handheld connected to the PC. We put the handheld in the cradle and
we can send and receive data over and over again without problem.

When we move the software to our client's system the handheld will do
the communication anywhere from 7 to 15 time successfully and then
fail with the error "No connection could be made because the target
machine actively refused it". We cannot repeat this in our simple
test environment and the client says they do not have any Firewalls
running on their internal network.

The handheld will authenticate against the server and request to do
one of 4 actions (Send, Receive Data, Receive Binary Updates, or
Transmit Application Logs for debug purposes). Each of these actions
start with an Initiation with the server. The basic initiation
protocol for this communication is as follows:

1. Handheld initiates communication with server on Specific Port
(45000 for example)
2. Server responds with new port for the handheld to communicate
with. This is a list kept internally and rolled over as needed, but
are assigned from some base port (46000) and assigned sequentially
over a specific range (200).
3. Handheld initiates communication with server on the new port and
sends or receives data based on user selection on the handheld.
(Option to send or receive)

We have run network traces using WireShark and have found that in the
failure case packets 1 and 2 go back and forth as expected. The
handheld sends Packet 3 to the server and the server never receives
the packet.

After this long-winded explanation, does anyone know a server or piece
of network hardware will block a TCP packet if the Source machine (The
PC with ActiveSync to the handheld) appears to be hitting a contiguous
set of ports one right after the other?

My theory, although I can't find anything to back me up yet, is that
some network switch or firewall software between the PC and server is
blocking the packets because it sees it as a possible "Port Scanning"
virus. Does this make sense?

Thanks in advance for your time.
-R-

 

[Jack Jackson]

[...]
After this long-winded explanation, does anyone know a server or piece
of network hardware will block a TCP packet if the Source machine (The
PC with ActiveSync to the handheld) appears to be hitting a contiguous
set of ports one right after the other?

My theory, although I can't find anything to back me up yet, is that
some network switch or firewall software between the PC and server is
blocking the packets because it sees it as a possible "Port Scanning"
virus. Does this make sense?

Absent any real data, it's certainly a viable theory. But I wouldn't
expect to see that sort of thing on an internal LAN. Seems more like an
ISP-level sort of thing.

That said, your description begs the question: why is the server rotating
through ports? This isn't typically something that is needed. If in fact
the connection resets are happening because of that behavior on the
server's part, it seems like you could just leave all the communications
on the one port and avoid the problem.

If nothing else, it might be worth temporarily changing your server so
that it only ever uses the initial "base port" and seeing if the problem
goes away. That might provide results faster than trying to track down
some hypothetical entity on the network that is resetting connection
attempts due to sequential port access.

Pete

Does the handheld always use the same port for its end of the
connection? If so, another possibility is TCP connection lifetime.
Once a TCP connection (the combination of source IP address,
destination IP address, source port and destination port) is closed,
that connection is unavailable for some period of time, possibly up to
2 minutes, to allow for any delayed packets to arrive.

 

[RobRasti]

I appreciate the feedback. The reason we rotate the ports is because
we have the possibility for over 150 handhelds possibly using this
server simultaneously to send and receive data. It is too much to
manage to have the handhelds assigned an individual port, and we
cannot have the primary port tied up in case another handheld connects
to send or receive data.

All that said, the main connection point for a large portion of the
user bank is a dial-up modem. When the connection is made through the
modem it does not exhibit this behavior. This is something I expect
to happen since the RAS server used resets the connection after the
modems hang up.

Thank you for all your input.
-R-

 

[RobRasti]

Thank you for the quick response.

Is the TCP Connection Lifetime something that is configurable on a
network? In our testing environment we have easily sent over 45
transmissions in this manner where in our client's production
environment it seems to exhibit this behavior after a handful of
attempts.

Either way, the handheld does not specify a port when it opens the
TCPClient object. It allows the handheld OS to dynamically open the
local port.

Thank you again for your help.
-R-

 

[Paul G. Tobey [eMVP]]

I'm afraid that I have to agree that this moving around of the port is
pointless. When you connect to a TCP port which is in the listening state,
if the server accepts your connection a new port *is* assigned, every time,
to the new client. It seems to me that you're just slowing down connections
for no good reason by having the server send back *another* listening port
number for the client to go off and connect to.

Paul T.

 

[Simon Hart [MVP]]

I would say this is a bad design. Usually when designing server processes you
listen on one port, when a request comes in you spawn a new thread to service
that client so that the original "listening" thread can continue to block and
accept connections.

How do you manage your firewall for exampe with your current system design.
This sounds a lot like how DCOM works which was a nightmare for admin staff.

 

[Jack Jackson]

I appreciate the feedback. The reason we rotate the ports is because
we have the possibility for over 150 handhelds possibly using this
server simultaneously to send and receive data. It is too much to
manage to have the handhelds assigned an individual port, and we
cannot have the primary port tied up in case another handheld connects
to send or receive data.

The server port won't be tied up. Multiple connections can be using
the same port on the server at the same time.

 

[RobRasti]

I want to thank all of you for your help. I was able to trace down
the issue to a connection attempt being made almost instantaneously
after closing a previous connection. This turned out to be a problem
with the local XP machine and the way it interfaced to with
ActiveSync. I guess the local port was not given time to cycle when
it hit a larger network with more traffic. Adding a LingerOption to
the TCPClient of roughly 5 seconds has cleared up the issue.

Again thank you all for your time.

-R-