multithreaded tcp/ip server large page file

[cj]

I have a 2005 TCP/IP server that creates a new thread to handle each
incoming TCP/IP request. Once the request has been answered by the
thread the TCP/IP socket is disconnected and the sub/thread ends. This
program originally written in 2003 and I rewrote it in 2005 earlier this
year. I don't know how long this has been going on but we just noticed
today that the program was using over a gig of virtual memory. Real
memory was quite low. The program had been running about 2 weeks.
Shutting it down and restarting it dropped the virtual memory to almost
nothing.

I don't know what causes this problem but I've always wondered about the
threads that end not being cleaned up. I use thread.start to start them
but nothing to end them. I've been told VB will clean these up.

One thing that didn't change over from 2003 to 2005 was
Thread.ApartmentState = ApartmentState.STA. 2005 says it obsolete but
the program runs so I ignored it's warning. Could this be causing a
problem?

Any ideas?

 

[cj]

My research suggest that there is no end thread type statement
necessary--just like I've been told. When the sub that comprises my
thread ends the thread ends. As ApartmentState.STA is obsolete but not
illegal I doubt it is causing virtual memory to grow. What else could
it be?

 

[Chris Mullins [MVP]]

Well, handling a request with a new thread is a pretty bad way to ago.

Each time you spin up a new thread, it's going to take up at least 1 meg of
fully committed virtual memory. You can read all the gory details about this
at:
http://www.bluebytesoftware.com/blog/PermaLink,guid,733d7537-f982-4886-af62-66bed0f97ab5.aspx

You really, really need to use a different model is you want your
application to scale in any meaningful way. Async I/O is really your only
long term solution.

Typically, you don't need to shutdown threads - they just go away when their
top level method exits. If you poke at your application, use a Performance
Counter to tell how many threads you've got running. If the number is very
high, then something is keeping the threads heald open. Do NOT (don't!
really!) just call "Kill Thread" on the open threads if this is the case -
figure out what the problem is and fix it.

You don't need to worry about the Apartment state stuff - it's not relevant
for a socket app.

 

[Charles Wang[MSFT]]

Hi cj,
This issue might be caused by creating a new thread to handle each incoming
request since one new thread will consume 1MB virtual memory. You may
consider to use Thread Pool to redesign your application. I would like to
provide the following articles here for your reference:
Programming the Thread Pool in the .NET Framework
http://msdn2.microsoft.com/en-us/library/ms973903.aspx
The Managed Thread Pool
http://msdn2.microsoft.com/en-us/library/0ka9477y.aspx

Hope this helps. Please feel free to let us know if you have any other
questions or concerns.
Have a good day!

Best regards,
Charles Wang
Microsoft Online Community Support
=====================================================
Get notification to my posts through email? Please refer to:
http://msdn.microsoft.com/subscriptions/managednewsgroups/default.aspx#notif
ications

If you are using Outlook Express, please make sure you clear the check box
"Tools/Options/Read: Get 300 headers at a time" to see your reply promptly.


Note: The MSDN Managed Newsgroup support offering is for non-urgent issues
where an initial response from the community or a Microsoft Support
Engineer within 1 business day is acceptable. Please note that each follow
up response may take approximately 2 business days as the support
professional working with you may need further investigation to reach the
most efficient resolution. The offering is not appropriate for situations
that require urgent, real-time or phone-based interactions or complex
project analysis and dump analysis issues. Issues of this nature are best
handled working with a dedicated Microsoft Support Engineer by contacting
Microsoft Customer Support Services (CSS) at
http://msdn.microsoft.com/subscriptions/support/default.aspx.
======================================================
When responding to posts, please "Reply to Group" via
your newsreader so that others may learn and benefit
from this issue.
======================================================
This posting is provided "AS IS" with no warranties, and confers no rights.
======================================================

 

[cj]

Is this 1MB of virtual memory released when the thread ends?

I'd like to see some documentation on the statement that each new thread
consumes 1MB virtual memory. If you can point me to an article I'd
appreciate it.

 

[Chris Mullins [MVP]]

How weird. I wrote up a full reply, and I see it on Google, but not here:
http://groups.google.com/group/micr...hread/thread/f81c1c2591938e5/5c9c7a2df42f27a6

Anyway, the original reply was:

Well, handling a request with a new thread is a pretty bad way to ago.

Each time you spin up a new thread, it's going to take up at least 1 meg of
fully committed virtual memory. You can read all the gory details about this
at:
http://www.bluebytesoftware.com/blog/PermaLink,guid,733d7537-f982-488...


You really, really need to use a different model is you want your
application to scale in any meaningful way. Async I/O is really your only
long term solution.


Typically, you don't need to shutdown threads - they just go away when their
top level method exits. If you poke at your application, use a Performance
Counter to tell how many threads you've got running. If the number is very
high, then something is keeping the threads heald open. Do NOT (don't!
really!) just call "Kill Thread" on the open threads if this is the case -
figure out what the problem is and fix it.


You don't need to worry about the Apartment state stuff - it's not relevant
for a socket app.

 

[Charles Wang[MSFT]]

Hi cj,
How did you end your thread? Generally if a thread handle is closed, the
related virtual memory will be released.
I recommend that you use Process Explorer to monitor your application to
check if the threads handles were closed after they ended.
You can download the tool from:
http://www.microsoft.com/technet/sysinternals/utilities/ProcessExplorer.mspx

Open Process Explorer, click View->Show Lower Panel, select Lower Panel
View-> Handles.

You may check if the thread number is growing and the handles are not
closed.

Best regards,
Charles Wang
Microsoft Online Community Support
=====================================================
Get notification to my posts through email? Please refer to:
http://msdn.microsoft.com/subscriptions/managednewsgroups/default.aspx#notif
ications

If you are using Outlook Express, please make sure you clear the check box
"Tools/Options/Read: Get 300 headers at a time" to see your reply promptly.


Note: The MSDN Managed Newsgroup support offering is for non-urgent issues
where an initial response from the community or a Microsoft Support
Engineer within 1 business day is acceptable. Please note that each follow
up response may take approximately 2 business days as the support
professional working with you may need further investigation to reach the
most efficient resolution. The offering is not appropriate for situations
that require urgent, real-time or phone-based interactions or complex
project analysis and dump analysis issues. Issues of this nature are best
handled working with a dedicated Microsoft Support Engineer by contacting
Microsoft Customer Support Services (CSS) at
http://msdn.microsoft.com/subscriptions/support/default.aspx.
======================================================
When responding to posts, please "Reply to Group" via
your newsreader so that others may learn and benefit
from this issue.
======================================================
This posting is provided "AS IS" with no warranties, and confers no rights.
======================================================

 

[cj]

I'll spare you all the details but it's started from the class
"sessionclass" which has only one sub "ThreadMain" like this:

Dim oSession As SessionClass
Dim oThread As Thread
oThread = New Thread(AddressOf oSession.ThreadMain)
oThread.Start()

It's my understanding that when the sub ThreadMain ends the tread also
ends. This is true right? I have not heard of closing a thread handle.

The first thing that ThreadMain does is increment a counter and the last
thing is decrement the counter. The counter is another class which uses
synclock to keep the threads from conflicting with eachother. This
counter is read an displayed by a timer in the main program to show how
many threads are running at any given time. It usually runs between 1
and 3. Part of the processing being done by each thread is contacting a
sql server and if the sql server is having problems processing by this
program backs up and I've seen it with over 100 threads open before. It
handles it fine as the sql request time out this program returns and
error response to the TCP/IP client that the thread is handling and the
sub/tread ends. It just takes much longer to timeout than to get a
response from SQL most of the time so during those times the concurrent
thread count will be much higher.

 

[cj]

I'll spare you all the details but it's started from the class
"sessionclass" which has only one sub "ThreadMain" like this:

Dim oSession As SessionClass
Dim oThread As Thread
oThread = New Thread(AddressOf oSession.ThreadMain)
oThread.Start()

It's my understanding that when the sub ThreadMain ends the tread also
ends. This is true right? I have not heard of closing a thread handle.

The first thing that ThreadMain does is increment a counter and the last
thing is decrement the counter. The counter is another class which uses
synclock to keep the threads from conflicting with eachother. This
counter is read an displayed by a timer in the main program to show how
many threads are running at any given time. It usually runs between 1
and 3. Part of the processing being done by each thread is contacting a
sql server and if the sql server is having problems processing by this
program backs up and I've seen it with over 100 threads open before. It
handles it fine as the sql request time out this program returns and
error response to the TCP/IP client that the thread is handling and the
sub/tread ends. It just takes much longer to timeout than to get a
response from SQL most of the time so during those times the concurrent
thread count will be much higher.

 

[cj]

I'll spare you all the details but it's started from the class
"sessionclass" which has only one sub "ThreadMain" like this:

Dim oSession As SessionClass
Dim oThread As Thread
oThread = New Thread(AddressOf oSession.ThreadMain)
oThread.Start()

It's my understanding that when the sub ThreadMain ends the tread also
ends. This is true right? I have not heard of closing a thread handle.

The first thing that ThreadMain does is increment a counter and the last
thing is decrement the counter. The counter is another class which uses
synclock to keep the threads from conflicting with eachother. This
counter is read an displayed by a timer in the main program to show how
many threads are running at any given time. It usually runs between 1
and 3. Part of the processing being done by each thread is contacting a
sql server and if the sql server is having problems processing by this
program backs up and I've seen it with over 100 threads open before. It
handles it fine as the sql request time out this program returns and
error response to the TCP/IP client that the thread is handling and the
sub/tread ends. It just takes much longer to timeout than to get a
response from SQL most of the time so during those times the concurrent
thread count will be much higher.

 

[cj]

Thank you for your response. It's always been my belief that everyone
is free to have their own opinions on what is a good or bad way to write
a program and I respect your opinion, however from where I stand this is
indeed an entirely appropriate way to complete the task at hand.

The 1 meg of fully committed virtual memory the threads use should not
be kept after the threads end according to what I've seen/read. And if
I'm right then letting each thread have a meg is no problem.

 

[Charles Wang[MSFT]]

Hi cj,
Yes, your understanding is right. When the sub treadmain ends, the thread
also ends. Just not sure if your server has a heavy load of requests and a
single request handling finishes in a long time, in other words, your
server might work in a very busy condition. Although some threads were
released, new threads were generated. The speed of creating new thread was
faster than the speed of releasing old thread.

I recommend that you run perfmon.exe to monitor the counter "Thread Count"
and "Virtual Bytes" of your process to see if this issue was caused by
thread explosion.

Hope this helps. If you have any other questions or concerns, please feel
free to let me know.

Best regards,
Charles Wang
Microsoft Online Community Support
=====================================================
Get notification to my posts through email? Please refer to:
http://msdn.microsoft.com/subscriptions/managednewsgroups/default.aspx#notif
ications

If you are using Outlook Express, please make sure you clear the check box
"Tools/Options/Read: Get 300 headers at a time" to see your reply promptly.


Note: The MSDN Managed Newsgroup support offering is for non-urgent issues
where an initial response from the community or a Microsoft Support
Engineer within 1 business day is acceptable. Please note that each follow
up response may take approximately 2 business days as the support
professional working with you may need further investigation to reach the
most efficient resolution. The offering is not appropriate for situations
that require urgent, real-time or phone-based interactions or complex
project analysis and dump analysis issues. Issues of this nature are best
handled working with a dedicated Microsoft Support Engineer by contacting
Microsoft Customer Support Services (CSS) at
http://msdn.microsoft.com/subscriptions/support/default.aspx.
======================================================
When responding to posts, please "Reply to Group" via
your newsreader so that others may learn and benefit
from this issue.
======================================================
This posting is provided "AS IS" with no warranties, and confers no rights.
======================================================

 

[cj]

I'd like to use perfmon but I can't find anything in it. It's too much
info. Where do I find "Thread Count" and "Virtual Bytes"?

I go to the bottom and right click then add counters and can't figure
out what to add. I think I need to set performance object to .net clr
memory but then which items under that do I monitor? I can't find
anywhere to monitor how many threads are running. In task manager I am
monitoring threads running and virtual memory.

I don't have a clue what I'm looking at in perfmon.

 

[Charles Wang[MSFT]]

Hi cj,
Please refer to the following steps to add the counters:
1. Run perfmon.exe from command prompt;
2. Delete the original counters by clicking DELETE key on keyboard;
2. Click Add to add counters;
3. Select Process in the Performance object list;
4. Select your process name in the list of "Select instances from list";
5. Add Thread Count and Virtual Bytes from the list "Select counters from
list";

For the CLR memory, you can select it from "Performance object" list, but I
am not sure if it is a helpful counter since the thread objects are
unmanaged.

Hope this helps. If you have any other questions or concerns, please feel
free to let me know.

Best regards,
Charles Wang
Microsoft Online Community Support
=====================================================
Get notification to my posts through email? Please refer to:
http://msdn.microsoft.com/subscriptions/managednewsgroups/default.aspx#notif
ications

If you are using Outlook Express, please make sure you clear the check box
"Tools/Options/Read: Get 300 headers at a time" to see your reply promptly.


Note: The MSDN Managed Newsgroup support offering is for non-urgent issues
where an initial response from the community or a Microsoft Support
Engineer within 1 business day is acceptable. Please note that each follow
up response may take approximately 2 business days as the support
professional working with you may need further investigation to reach the
most efficient resolution. The offering is not appropriate for situations
that require urgent, real-time or phone-based interactions or complex
project analysis and dump analysis issues. Issues of this nature are best
handled working with a dedicated Microsoft Support Engineer by contacting
Microsoft Customer Support Services (CSS) at
http://msdn.microsoft.com/subscriptions/support/default.aspx.
======================================================
When responding to posts, please "Reply to Group" via
your newsreader so that others may learn and benefit
from this issue.
======================================================
This posting is provided "AS IS" with no warranties, and confers no rights.
======================================================

 

[cj]

Ok, I am tracking Virtual Bytes and Threads. I'm also tracking Private
bytes and working set--I read an article that suggested those. I'm
going to keep the graph updating ever 5 minutes and we'll see where the
trend leads today. I know the memory trend will be up. I doubt the
threads will go up. But anyway, we'll see where it goes and I'll let
you know.

Thanks

 

[cj]

Hi Charles, I responded to you via email so I could send you a picture
of the graph. For what it is worth to others I'm posting this here as well.

I've been running perfmon all day and this is what I have now.
Threadcount has fluctuated normally. I see no upward trend at all.
Virtual Bytes seems to be stepping up every few hours. I don't knwo
what to make of private bytes and working set as they are going up at
about a 45 degree angle--(they seem to indicate a problem but I don't
know what). This program has been running for over a year now without
any problems but the pc is scheduled to reboot weekly. We never looked
at the memory until now. The scheduled reboot got turned off and it
went two weeks and someone noticed an error on the screen that said we
were running out of virtual memory. That's what got us started looking
into this. The graph is set to update every 5 minutes.

Thanks for your help

 

[cj]

And I see the email I sent you came back undeliverable. Well there's
1001 ways to do things so I'll find another but I gotta get home to do that.

 

[cj]

Here's the picture:

http://img167.imageshack.us/my.php?image=mozscreenshot2mq9.jpg

 

[Charles Wang[MSFT]]

Hi cj,
The continuous increase in the Process\Private Bytes counter indicates a
possible memory leak. I recommend that you carefully check your code to see
if some unmanaged objects/resources were not released after they are used.

Also, you may refer to the section Step 2: Identify Workload of this
article:
Performance Testing for Application Blocks
http://msdn2.microsoft.com/en-us/library/ms998241.aspx

NOTE: This issue is memory related. To track the root cause, memory dump
analysis may be required, however this work can only be done by CSS. Our
managed newsgroup is focused on break/fix issues that are neither urgent
nor complex. If the issue is urgent to your business, it is recommended
that you contact Microsoft Customer Support Services (CSS) via telephone so
that a dedicated Support Professional can assist you in a more efficient
manner. Please be advised that contacting phone support will be a charged
call.

To obtain the phone numbers for specific technology request please take a
look at the web site listed below.
http://support.microsoft.com/default.aspx?scid=fh;EN-US;PHONENUMBERS

If you are outside the US please see http://support.microsoft.com for
regional support phone numbers.

Best regards,
Charles Wang
Microsoft Online Community Support
=====================================================
When responding to posts, please "Reply to Group" via
your newsreader so that others may learn and benefit
from this issue.
======================================================
This posting is provided "AS IS" with no warranties, and confers no rights.
======================================================

 

[cj]

Forgive me for asking a dumb question but what would be unmanaged
objects/resources? I thought all .net programs were managed.