Connection limit

[cj2]

I still think I'm hitting some limit on connections from a windows app
on Windows XP to a web service on Windows Server 2008. I have read XP
has a 10 connection limit of some kind. I need to know more about this.

I've asked before but have trouble explaining. I'll try again.

The main form loops every 200 milliseconds:
launches a thread
displays the counter class's total concurrent threads value


Each thread:
notes the time it starts
increments a counter class
Calls a web service (same info each time)
notes the time it finished and any errors that occured
write the times etc to a log file
decrements a counter class
thread ends.

The web service records the time it receives a request and the time it
receives a request and the total time it took to process it. it's
always between .1 and .3 seconds.

What I see. I start the program about 9:42. The concurrent threads
starts to climb it eventually hit 2000 connections! I aborted the
program at that point at about 9:50 as it was returning out of memory
and unable to start a new thread messages. After aborting the program
the threads remained (as they should) I noted this in task mgr too.
They slowly came down to 0 about 10:10. Through this whole process the
web service says every request it got took an average of .283 seconds to
process and it got requests from 9:42 to 10:05. The program calling the
web service on the other hand reports it made requests from 9:42 to
9:50. the first few were fast but then they started taking longer and
longer (up to 15 minutes) for the threads to finish. The thread's sole
purpose is to call that web service sending the same value to it (for
testing) and get the result.

Strangely I tried repeating the test and this time it worked fine
staying at 1 or 2 concurrent connections for a whole hour of testing.

This is quit like in real life with real requests. I swear that the
problem is that once it hits a certain number of concurrent threads
running it reaches some sort of log jam and request are then being
queued and released to the web service slowly.

 

[cj2]

I have one additional statement to make. While it did hit 2000
concurrent connections in that one test. Problems start when the
concurrent connections go somewhere between 10 and 25. It's at those
times when the web service says it took .28 seconds to reply and the
calling program says it took 15 seconds to get the reply. That's a
problem as we need responses in less than a second. 2000 and 15 minutes
is an extreme example.

 

[Colbert Zhou [MSFT]]

Hi,

I think this is not related to the XP concurrent connection limitation.
XP's concurrent connections limitation is 256 for TCP/IP connections, as
far as I know, and when you exceed the limitation, you will receive "TCP/IP
has reached the security limit imposed on the number of concurrent TCP
connect attempts".

I am trying to understand more about your test scenario. From my current
understanding, I do not think the test is similar with a real life
condition. Actually, which peformance we really need to concern is the
server-side performance, right? In this case, the server side webserivce
has a good performance and it always process and reply in less than one
second. The problem is the client's repsonse time is too long to accept
from the log's perspective, right? My concern is that in real life, the
clients that make request calling may be in different computers and even in
different places. But in our test, we put 2000 clients in one process at
one machine. The test way itself will slow the client application because
when we create a thread, the process will allocate a memory snippet for
that thread stack. 2000 threads means that 2000*1M = 2G virtual memory
space. It definitly slow the application. And in each of the thread, we
make loggs to the local drivers. That also cause a lot of time to do IO
operations. So when there are too many threads running together in one
process. The log operation will be very slow. So you will see your clients
too slow. The right way to test in this case is letting different physical
clients consume your webserivce at one time. I believe, as long as your
webserivce reply in one second, all of your clients will get the reply
soon. The performance issue should not exist in real life.

Hope this gives the right direction. Have a nice day!

Best regards,
Colbert Zhou
Microsoft Newsgroup Support Team

 

[cj2]

Actually. in this case it is close to reality as we do have one
application that repeatedly calls the web service. The web service is
also handling other requests as well from various sources but over half
of it's calls come from this one app. In the production app the threads
are answering TCP/IP connections taking the request and sending it to
the web service then taking the web services response and sending it
back to the TCP/IP client. It allows old systems with code out of our
control to talk to the web service. For the sake of testing we've taken
the TCP/IP part completely out and just have the test program simulate
the load it would be sending the web service by creating threads that
call the web service and receive it's response.

Again while I mention that the test app ran up to 2000 concurrent
threads this was an extreme example and we normally see responses taking
too long when we get somewhere around 10 to 25 concurrent threads.

In the web service I time from as close to the top of the code as
possible to right before I write the total time it took into the sql
log. It averages .283 seconds. In the calling app I note the time
right before the the line where I call the web service and again right
after then write it to a text file. It says it took minutes. From my
perspective the calling app makes a call but it takes minutes before the
web service sees it then the web service processed in .283 seconds and
returned the result. Or the delay is on the other end. The calling app
makes the request and the web service processes it in .283 seconds then
the return takes minutes.

Or the writing of the total times by the web app to sql is taking
forever as I have to take the end time right before the write to the sql
log which is of course right before the web service returns the results.
But if writing to sql was the problem I would think we would see
problems with other systems here that are using sql during these times
and we haven't.

Regardless I trap and log all errors and haven't gotten any. The
process just takes too long.

I know this is a very hard thing to diagnose but if you know of any
limitations I might be running into let me know. I am just looking for
other ideas. I'm still testing.

 

[Colbert Zhou [MSFT]]

Hi ,

If we are using TCP/IP connection in your XP, yes, we have a 256 connection
limitations. You can refer the following link to remove this limitation.
But I do not think the described issue is related to this because you
mention that 10-25 cocurrent threads also causes this problem.
http://www.tech-faq.com/concurrent-connections-limit-in-windows-xp.shtml

Actually, we can use the tool fiddler to listen and trace your client. The
tool can trace all out connections and log the requesting and reponse time.
Please download the fiddler from the following page,
http://www.fiddler2.com/fiddler2/
http://msdn.microsoft.com/en-us/library/bb250442(VS.85).aspx

So that we can detect where the performance bottle-neck is, the client app
itself or the communication time.


Best regards,
Colbert Zhou
Microsoft Newsgroup Support Team

 

[cj2]

A fellow at the office here has been quite helpful today. He found
something on the web that says Windows XP SP2 and up limit you to 10
simultaneous HTTP connections to the same web server. As a web service
call is HTTP traffic and our testing has been from XP SP3 to of course
the same server and about 10 is where I was seeing the problem this
looks promising.

So that means I do need to test my app running on a server. It'll be
Monday before I can get to this as my afternoon is booked but I'll post
back results when I have some.

 

[cj2]

I've downloaded and trying to use fiddler2. What exactly should I be
looking at?

 

[Colbert Zhou]

Hi Chris,

When we open fiddler, the fiddler will listen to all outgoing and incoming
connections and record all important statistics for us. For example, when I
consume a remote web service, I can see fiddler record the following
connections,

# Result Protocol Host URL Body Caching Content-Type Process Comments Custom
0 200 HTTP www.xignite.com
/xMetals.asmx/GetLastRealTimeMetalQuote?Type=XAU&Currency=USD 692 private,
max-age=0 text/xml; charset=utf-8 iexplore:1332

And when I select this recorded item, I can see statistics in the right
panel. For example, in the Statistics tab, I see,
----------------------------------
Request Count: 1
Bytes Sent: 889
Bytes Received: 967

ACTUAL PERFORMANCE
--------------
Requests started at: 18:22:53:0702
Responses completed at: 18:22:54:2832
Aggregate Session time: 00:00:01:2130
Sequence (clock) time: 00:00:01.2130000
TCP/IP Connect time: 46ms

RESPONSE CODES
--------------
HTTP/200: 1

RESPONSE BYTES (by Content-Type)
--------------
~headers: 275
text/xml: 692

ESTIMATED WORLDWIDE PERFORMANCE
--------------
The following are VERY rough estimates of download times when hitting
servers based in WA, USA.
........

--------------------------------
So we can see the request time and response time as well as the process time
used in this request. If you use fiddler in your side, I think we can find
get some clues from there about what causes the performance issue.


Best regards,
Colbert Zhou

 

[cj2]

It would appear my delay is before the http request goes through
fiddler. The threads appears to get to line where they call the web
service and then they pause for no reason.

If I start the program generating threads every 200 milliseconds each
wanting to call the web service. Sometimes these threads back up for no
apparent reason and there will be for example 50 threads concurrently
running each trying to make web request. If I have the program stop
creating threads at that point, I see in Fiddler that it takes over a
minute for those all of those threads to make their http calls to the
web service, get their responses and end. Keep in mind the web service
is responding in less that a second to every request it gets.

Where would this "queuing" be happening??? It is happening whether
program is running on a Windows Server 2008 machine or on my XP machine.

The worst part is sometimes the threads never back up and we never get
more than 1 or 2 threads running concurrently so it's not very predictable.

The web server and the web service are not being used by anyone else at
this time.

Hi Chris,

When we open fiddler, the fiddler will listen to all outgoing and
incoming connections and record all important statistics for us. For
example, when I consume a remote web service, I can see fiddler record
the following connections,

# Result Protocol Host URL Body Caching Content-Type Process Comments
Custom
0 200 HTTP www.xignite.com
/xMetals.asmx/GetLastRealTimeMetalQuote?Type=XAU&Currency=USD 692
private, max-age=0 text/xml; charset=utf-8 iexplore:1332

And when I select this recorded item, I can see statistics in the right
panel. For example, in the Statistics tab, I see,
----------------------------------
Request Count: 1
Bytes Sent: 889
Bytes Received: 967

ACTUAL PERFORMANCE
--------------
Requests started at: 18:22:53:0702
Responses completed at: 18:22:54:2832
Aggregate Session time: 00:00:01:2130
Sequence (clock) time: 00:00:01.2130000
TCP/IP Connect time: 46ms

RESPONSE CODES
--------------
HTTP/200: 1

RESPONSE BYTES (by Content-Type)
--------------
~headers: 275
text/xml: 692

ESTIMATED WORLDWIDE PERFORMANCE
--------------
The following are VERY rough estimates of download times when hitting
servers based in WA, USA.
.......

--------------------------------
So we can see the request time and response time as well as the process
time used in this request. If you use fiddler in your side, I think we
can find get some clues from there about what causes the performance issue.


Best regards,
Colbert Zhou

I've downloaded and trying to use fiddler2. What exactly should I be
looking at?

A fellow at the office here has been quite helpful today. He found
something on the web that says Windows XP SP2 and up limit you to 10
simultaneous HTTP connections to the same web server. As a web
service call is HTTP traffic and our testing has been from XP SP3 to
of course the same server and about 10 is where I was seeing the
problem this looks promising.

So that means I do need to test my app running on a server. It'll be
Monday before I can get to this as my afternoon is booked but I'll
post back results when I have some.


Colbert Zhou [MSFT] wrote:
Hi ,

If we are using TCP/IP connection in your XP, yes, we have a 256
connection limitations. You can refer the following link to remove
this limitation. But I do not think the described issue is related
to this because you mention that 10-25 cocurrent threads also causes
this problem.
http://www.tech-faq.com/concurrent-connections-limit-in-windows-xp.shtml


Actually, we can use the tool fiddler to listen and trace your
client. The tool can trace all out connections and log the
requesting and reponse time. Please download the fiddler from the
following page,
http://www.fiddler2.com/fiddler2/
http://msdn.microsoft.com/en-us/library/bb250442(VS.85).aspx

So that we can detect where the performance bottle-neck is, the
client app itself or the communication time.


Best regards,
Colbert Zhou
Microsoft Newsgroup Support Team

 

[Colbert Zhou [MSFT]]

Hi Chris,

When you see the client application threads back up, what is the status of
the CPU usage of that client process? If the CPU usage is high usage, I
think maybe the CPU is so busy and some of threads really do not get a
chance to be executed. That is to say the issue should cause from the
codes, instead of a system connection limitation(or say a queue). We can
use the Process Monitor to observe all threads calling stacks. So we may
get some clues from the threads calling stack.

And for better support on this, would you mind sharing your client
application project? If we can reproduce the issue in our side, we can do
future investigation and try to help on this. If that is possible, you can
contact me via (e-mail address removed) (remove online.)

Have a nice day!


Best regards,
Colbert Zhou
Microsoft Online Support Team

 

[cj2]

I responded to your email address. I can't really post my code here but
I zipped the project and attached it to the email.

As noted in the email my cpu usage was 4%.