multithreaded tcp/ip server large page file

[Rory Becker]

Forgive me for asking a dumb question but what would be unmanaged

objects/resources? I thought all .net programs were managed.

In .Net, Memory is managed, but filehandles, COM Objects and other things
are not.

 

[cj]

But memory is the issue here (memory leak). Are you saying some memory
is managed and some is not?

 

[cj]

Maybe this is a better use of our time--to make sure I'm ending objects
correctly.

Do you see anything wrong with this class? It called by all my threads
whenever they need to log a transaction.

Public Class MyStringLogger
Private Shared m_loglock As New Object

Public Shared Sub Write(ByVal fileName As String, ByVal strToWrite
As String)
SyncLock (m_loglock)
Try
Dim sw As New System.io.StreamWriter(fileName, True)
sw.WriteLine(strToWrite)
sw.Close()
Catch
'if it fails forget it--not really that important
anyway
End Try
End SyncLock
End Sub
End Class

Do I need sw.dispose() after sw.close()?

I've always wondered how to ensure it's closed in case an error occurs
in opening the file or writing to it etc. In the case above if it fails
to write the file it would be left open I suppose--then again since it
is declared inside the try catch block it should cease to exist when
that section of code is done executing. Right? The network might be
down and the file can't be written to so I have to plan for this stuff.

 

[Charles Wang[MSFT]]

Hi cj,
Generally for COM, unmanaged DLL, pointers, and file handles etc, they are
unmanaged. For .NET CLR objects, they are managed.

For managed and unmanaged code interoperability, you can refer to:
An Overview of Managed/Unmanaged Code Interoperability
http://msdn2.microsoft.com/en-us/library/ms973872.aspx

Unmanaged code may cause memory leak however even managed code may also
probably cause the issue which appears to have a memory leak. I recommend
that you refer to this article:
How to identify memory leaks in the common language runtime
http://support.microsoft.com/kb/318263

You may refer to the KB article and check if there is similar issue in your
code.
Please feel free to let me know if you have any other questions or
concerns. I am glad to work with you for further research.

Best regards,
Charles Wang
Microsoft Online Community Support
=====================================================
Get notification to my posts through email? Please refer to:
http://msdn.microsoft.com/subscriptions/managednewsgroups/default.aspx#notif
ications

If you are using Outlook Express, please make sure you clear the check box
"Tools/Options/Read: Get 300 headers at a time" to see your reply promptly.


Note: The MSDN Managed Newsgroup support offering is for non-urgent issues
where an initial response from the community or a Microsoft Support
Engineer within 1 business day is acceptable. Please note that each follow
up response may take approximately 2 business days as the support
professional working with you may need further investigation to reach the
most efficient resolution. The offering is not appropriate for situations
that require urgent, real-time or phone-based interactions or complex
project analysis and dump analysis issues. Issues of this nature are best
handled working with a dedicated Microsoft Support Engineer by contacting
Microsoft Customer Support Services (CSS) at
http://msdn.microsoft.com/subscriptions/support/default.aspx.
======================================================
When responding to posts, please "Reply to Group" via
your newsreader so that others may learn and benefit
from this issue.
======================================================
This posting is provided "AS IS" with no warranties, and confers no rights.
======================================================

 

[cj]

Charles, I've been reading and I don't get the managed vs unmanaged
because first I read that all .net code is managed then I keep seeing
where everything I use is unmanaged. File operations and even sql
connections. Anyway I just read an article that said if the object has
a .dispose method use it. That makes sense to me. Given that advise
I'm going to look over my code and see what I can do. I'm going to
consider this thread finished as I think my questions on .dispose can
stand by themselves outside of this memory leak issue. Thanks for all
your help.

 

[Tom Shelton]

Maybe this is a better use of our time--to make sure I'm ending objects
correctly.

Do you see anything wrong with this class? It called by all my threads
whenever they need to log a transaction.

Public Class MyStringLogger
Private Shared m_loglock As New Object

Public Shared Sub Write(ByVal fileName As String, ByVal strToWrite
As String)
SyncLock (m_loglock)
Try
Dim sw As New System.io.StreamWriter(fileName, True)
sw.WriteLine(strToWrite)
sw.Close()
Catch
'if it fails forget it--not really that important
anyway
End Try
End SyncLock
End Sub
End Class

Do I need sw.dispose() after sw.close()?

No. sw.Close calls dispose internally. In fact, the implementation
is:

Public Sub Close ()
Me.Dispose(True)
GC.SupressFinalize(Me)
End Sub

CJ, you might want to look at VB.NET's USING statement. It makes a
lot of this stuff much easier, since it guarentees that Dispose will
be called - even if you exit via an exception. I would probably
implement your above block somehting like:

SyncLock (m_loglock)
Try
Using sw As New StreamWriter(fileName, True)
sw.WriteLine(strToWrite)
End Using ' automatically closed
Catch
End Try
End SyncLock

Anyway, HTH

 

[cj]

I'll check into it.

No. sw.Close calls dispose internally. In fact, the implementation
is:

Public Sub Close ()
Me.Dispose(True)
GC.SupressFinalize(Me)
End Sub

CJ, you might want to look at VB.NET's USING statement. It makes a
lot of this stuff much easier, since it guarentees that Dispose will
be called - even if you exit via an exception. I would probably
implement your above block somehting like:

SyncLock (m_loglock)
Try
Using sw As New StreamWriter(fileName, True)
sw.WriteLine(strToWrite)
End Using ' automatically closed
Catch
End Try
End SyncLock

Anyway, HTH

 

[Charles Wang[MSFT]]

Hi cj,
Yes, it is wisdom to use Dispose method to release resources.

Not sure if you noticed the description in the KB article 318263:
=============================================
Additionally, a project may only appear to have a memory leak. This
condition can occur if many large objects (such as DataTable objects) are
declared and then added to a collection (such as a DataSet). The resources
that these objects own may never be released, and the resources are left
alive for the whole run of the program. This appears to be a leak, but
actually it is just a symptom of the way that memory is being allocated in
the program.
=============================================

All objects that implement IDispose interface have a Dispose method.
Dataset implement IDispose interface, so you can call explicitly call it to
release resources. Otherwise, you will wait for GC releasing the objects.,
however we do not know when GC collect the memory. For those temorporay
objects, such as returned large objects, please explicityly call Dispose
method to release resources after you use them.

Best regards,
Charles Wang
Microsoft Online Community Support
=====================================================
Get notification to my posts through email? Please refer to:
http://msdn.microsoft.com/subscriptions/managednewsgroups/default.aspx#notif
ications

If you are using Outlook Express, please make sure you clear the check box
"Tools/Options/Read: Get 300 headers at a time" to see your reply promptly.


Note: The MSDN Managed Newsgroup support offering is for non-urgent issues
where an initial response from the community or a Microsoft Support
Engineer within 1 business day is acceptable. Please note that each follow
up response may take approximately 2 business days as the support
professional working with you may need further investigation to reach the
most efficient resolution. The offering is not appropriate for situations
that require urgent, real-time or phone-based interactions or complex
project analysis and dump analysis issues. Issues of this nature are best
handled working with a dedicated Microsoft Support Engineer by contacting
Microsoft Customer Support Services (CSS) at
http://msdn.microsoft.com/subscriptions/support/default.aspx.
======================================================
When responding to posts, please "Reply to Group" via
your newsreader so that others may learn and benefit
from this issue.
======================================================
This posting is provided "AS IS" with no warranties, and confers no rights.
======================================================

 

[cj]

I did notice that but I really think this is some form of leak. I've
got some more testing I'm running tonight. If this doesn't work I'll
have to figure out the using statement I guess.

 

[Charles Wang[MSFT]]

Hi cj,
Thanks for your updating and response.

Just feel free to post back if you encounter any problems in future. Have a
good day!

Best regards,
Charles Wang
Microsoft Online Community Support
=====================================================
Get notification to my posts through email? Please refer to:
http://msdn.microsoft.com/subscriptions/managednewsgroups/default.aspx#notif
ications

If you are using Outlook Express, please make sure you clear the check box
"Tools/Options/Read: Get 300 headers at a time" to see your reply promptly.


Note: The MSDN Managed Newsgroup support offering is for non-urgent issues
where an initial response from the community or a Microsoft Support
Engineer within 1 business day is acceptable. Please note that each follow
up response may take approximately 2 business days as the support
professional working with you may need further investigation to reach the
most efficient resolution. The offering is not appropriate for situations
that require urgent, real-time or phone-based interactions or complex
project analysis and dump analysis issues. Issues of this nature are best
handled working with a dedicated Microsoft Support Engineer by contacting
Microsoft Customer Support Services (CSS) at
http://msdn.microsoft.com/subscriptions/support/default.aspx.
======================================================
When responding to posts, please "Reply to Group" via
your newsreader so that others may learn and benefit
from this issue.
======================================================
This posting is provided "AS IS" with no warranties, and confers no rights.
======================================================

 

[Charles Wang[MSFT]]

Hi cj,
How about this issue?
If you need any assistance, please feel free to post back.

Have a good day!

Charles Wang
Microsoft Online Community Support

======================================================
When responding to posts, please "Reply to Group" via
your newsreader so that others may learn and benefit
from this issue.
======================================================
This posting is provided "AS IS" with no warranties, and confers no rights.
======================================================

 

[cj]

Hi Charles,

I can't say that I've seen any improvement. Private Bytes and Working
Set climb steadily during the day when the system is busy. Virtual
Bytes slowly climbs. It'll remain flat for hours then jump up. Threads
have shown no signs of increase. I'm using dispose for everything that
offers it now. I haven't had time to look into Using yet.

Thanks for checking,
cj

 

[Charles Wang[MSFT]]

Hi cj,
In this case, I recommend that you also add the following counters to
monitor your .NET application to see if the CLR memory is being used highly:
..NET CLR Memory\# Bytes in All Heaps
..NET CLR Memory\Large Object Heap Size
..NET CLR Memory\% Time in GC

Please feel free to post back once you collect the data. I am glad to work
with you for further assistance.
Have a nice day!

Best regards,
Charles Wang
Microsoft Online Community Support
=====================================================
Get notification to my posts through email? Please refer to:
http://msdn.microsoft.com/subscriptions/managednewsgroups/default.aspx#notif
ications

If you are using Outlook Express, please make sure you clear the check box
"Tools/Options/Read: Get 300 headers at a time" to see your reply promptly.


Note: The MSDN Managed Newsgroup support offering is for non-urgent issues
where an initial response from the community or a Microsoft Support
Engineer within 1 business day is acceptable. Please note that each follow
up response may take approximately 2 business days as the support
professional working with you may need further investigation to reach the
most efficient resolution. The offering is not appropriate for situations
that require urgent, real-time or phone-based interactions or complex
project analysis and dump analysis issues. Issues of this nature are best
handled working with a dedicated Microsoft Support Engineer by contacting
Microsoft Customer Support Services (CSS) at
http://msdn.microsoft.com/subscriptions/support/default.aspx.
======================================================
When responding to posts, please "Reply to Group" via
your newsreader so that others may learn and benefit
from this issue.
======================================================
This posting is provided "AS IS" with no warranties, and confers no rights.
======================================================

 

[cj]

Hi Charles,

I added the counters yesterday and it's been almost 24 hours now.
# Bytes in All Heaps fluctuates but has shown no signs of increase
Last 585284, Avg 565394, Min 399088, Max 860336
Large Object Heap Size hasn't changed since it started
Last 33816, Avg 33816, Min 33816, Max 33816
% Time in GC has fluctuated but is very small
Last .002, Avg .004, Min .001, Max .016

Private bytes continues to climb whever the system is active
Last 117538816, Avg 88500153, Min 25710592, Max 120004608
Working Set also continues to climb at the same rate as private bytes
Last 119078912, Avg 89172462, Min 25686016, Max 119353344
Virtual Bytest continues to climb in a stair step fashion--jumps up
every few hours.
Last 294690816, Avg 254321252, Min 161128448, Max 300130304
Thread Count shows no signs of increase
Last 16, Avg 15, Min 13, Max 18

I still need to look at Using. I've been busy on other things. This
doesn't cause a problem unless it goes over a week and normally it's
rebooted weekly. But I'm very curious as to what is going on and I
really appreciate you help.

 

[Charles Wang[MSFT]]

Hi cj,
Did you have some P/Invoke or COM interop in your code? Also, please checK
if there are some unmanaged resources such as bitmap, connections,
Filestream etc which may not be disposed after they are used.
I recommend that you refer to this article before you make further checking
to your code:
Garbage Collection: Automatic Memory Management in the Microsoft .NET
Framework
http://msdn.microsoft.com/msdnmag/issues/1100/gci/

Please feel free to let me know if you have any other questions or
concerns.

Best regards,
Charles Wang
Microsoft Online Community Support
=====================================================
Get notification to my posts through email? Please refer to:
http://msdn.microsoft.com/subscriptions/managednewsgroups/default.aspx#notif
ications

If you are using Outlook Express, please make sure you clear the check box
"Tools/Options/Read: Get 300 headers at a time" to see your reply promptly.


Note: The MSDN Managed Newsgroup support offering is for non-urgent issues
where an initial response from the community or a Microsoft Support
Engineer within 1 business day is acceptable. Please note that each follow
up response may take approximately 2 business days as the support
professional working with you may need further investigation to reach the
most efficient resolution. The offering is not appropriate for situations
that require urgent, real-time or phone-based interactions or complex
project analysis and dump analysis issues. Issues of this nature are best
handled working with a dedicated Microsoft Support Engineer by contacting
Microsoft Customer Support Services (CSS) at
http://msdn.microsoft.com/subscriptions/support/default.aspx.
======================================================
When responding to posts, please "Reply to Group" via
your newsreader so that others may learn and benefit
from this issue.
======================================================
This posting is provided "AS IS" with no warranties, and confers no rights.
======================================================

 

[cj]

I do have a COM control that does the TCP/IP communication. I also
write to files and access a SQL db in this program. As a result of
seeing the Memory usage climb I've started calling .dispose for all
objects I use that implement it. For example the class that writes to
the files--the only class in this prg that writes to files.

Public Class MyStringLogger
Private Shared m_loglock As New Object

Public Shared Sub Write(ByVal fileName As String, ByVal
strToWrite As String)
SyncLock (m_loglock)
'remark out this try catch block to prevent all writing
to disk
Try
Dim sw As New System.io.StreamWriter(fileName, True)
sw.WriteLine(strToWrite)
sw.Close()
sw.Dispose()
Catch
End Try
End SyncLock
End Sub
End Class

My SQL connections are closed and disposed as soon as they are no longer
needed.

I've contacted Catalyst the COM provider and they advised me to use an
uninitialize method which I now use. They do not provide a dispose method.

 

[Charles Wang[MSFT]]

Hi cj,
Sorry for delaying this response due to a sick leave.

If your SQL connections are closed and disposed as soon as they are no
longer needed, they won't be leaked. For the COM objects or references, you
may check if they provide a method to release themselves. Dispose method is
just a .NET class who implemented IDispose interface, however for those
unmanaged objects, if they were not wrapped in a .NET class which
implemented IDispose interface, you need to use their self-provided release
function to release themselves.

From the perfmon monitor result, we can see that this is a unmanged
resources leakage issue. The managed resources were controlled very well in
your application. I do not know what the COM component you used was, but
there might be another possible reason, if the COM component itself has
memory leaks, it is hard for you to correct it in your .NET application.

I recommend that you work with your COM developers to identify which COM
objects and references need to be released in what way.

Hope this helps. If you have any other questions or concerns, please feel
free to post back. I am glad to work with you for further assistance.

Best regards,
Charles Wang
Microsoft Online Community Support
=====================================================
Get notification to my posts through email? Please refer to:
http://msdn.microsoft.com/subscriptions/managednewsgroups/default.aspx#notif
ications

If you are using Outlook Express, please make sure you clear the check box
"Tools/Options/Read: Get 300 headers at a time" to see your reply promptly.


Note: The MSDN Managed Newsgroup support offering is for non-urgent issues
where an initial response from the community or a Microsoft Support
Engineer within 1 business day is acceptable. Please note that each follow
up response may take approximately 2 business days as the support
professional working with you may need further investigation to reach the
most efficient resolution. The offering is not appropriate for situations
that require urgent, real-time or phone-based interactions or complex
project analysis and dump analysis issues. Issues of this nature are best
handled working with a dedicated Microsoft Support Engineer by contacting
Microsoft Customer Support Services (CSS) at
http://msdn.microsoft.com/subscriptions/support/default.aspx.
======================================================
When responding to posts, please "Reply to Group" via
your newsreader so that others may learn and benefit
from this issue.
======================================================
This posting is provided "AS IS" with no warranties, and confers no rights.
======================================================

 

[Charles Wang[MSFT]]

Hi cj,
What is going on with this issue? Please feel free to post back if you
encounter any issues.
Have a nice day!

Best regards,
Charles Wang
Microsoft Online Community Support
=====================================================
Get notification to my posts through email? Please refer to:
http://msdn.microsoft.com/subscriptions/managednewsgroups/default.aspx#notif
ications

If you are using Outlook Express, please make sure you clear the check box
"Tools/Options/Read: Get 300 headers at a time" to see your reply promptly.


Note: The MSDN Managed Newsgroup support offering is for non-urgent issues
where an initial response from the community or a Microsoft Support
Engineer within 1 business day is acceptable. Please note that each follow
up response may take approximately 2 business days as the support
professional working with you may need further investigation to reach the
most efficient resolution. The offering is not appropriate for situations
that require urgent, real-time or phone-based interactions or complex
project analysis and dump analysis issues. Issues of this nature are best
handled working with a dedicated Microsoft Support Engineer by contacting
Microsoft Customer Support Services (CSS) at
http://msdn.microsoft.com/subscriptions/support/default.aspx.
======================================================
When responding to posts, please "Reply to Group" via
your newsreader so that others may learn and benefit
from this issue.
======================================================
This posting is provided "AS IS" with no warranties, and confers no rights.
======================================================

 

[cj]

Sorry Charles, other priorities came up and I haven't been able to work
on this for awhile now. I did add using around the SQL and file handles
but I encountered an unhanded exception somewhere after 8-10 hours of
running. I don't know what the cause was but since the memory was still
climbing anyway I changed the code back to before I added using.

I'd already spoken with the com vendor and he told me to make sure I
called .Uninitialize. I am but that doesn't help.

I don't like knowing it's using increasing amounts of memory and I can't
explain it but it runs incredibly well as long as the program is
restarted at least once a week. Thankfully that isn't a problem as when
we first wrote the program we had worried about the ability of the pc to
run a multi threaded app for weeks at a time w/o a reboot and the
program was written to reboot the pc on selected days at 2:00am. This
"memory leak" might have been occurring all along but nobody noticed
until the automatic reboot was turned off and it went just over 2 weeks
an gave the memory error.

I appreciate your help but I don't know what more we can do. As I have
time I'll keep trying things.

 

[Charles Wang[MSFT]]

Hi cj,
I appreciate your updating and feedback.

Since this issue is complex and may need dump analysis for tracking the
root cause, I would like to recommend a senior support channel for you.
Effectively and immediately you can contact Microsoft Customer Support
Services (CSS) via telephone so that a dedicated Support Professional can
assist you in a more efficient manner. Please be advised that contacting
phone support will be a charged call.

To obtain the phone numbers for specific technology request please take a
look at the web site listed below.
http://support.microsoft.com/default.aspx?scid=fh;EN-US;PHONENUMBERS

If you are outside the US please see http://support.microsoft.com for
regional support phone numbers.

If you have any other questions or concerns, please feel free to let me
know.
Have a good day!

Best regards,
Charles Wang
Microsoft Online Community Support
=====================================================
Get notification to my posts through email? Please refer to:
http://msdn.microsoft.com/subscriptions/managednewsgroups/default.aspx#notif
ications

If you are using Outlook Express, please make sure you clear the check box
"Tools/Options/Read: Get 300 headers at a time" to see your reply promptly.


Note: The MSDN Managed Newsgroup support offering is for non-urgent issues
where an initial response from the community or a Microsoft Support
Engineer within 1 business day is acceptable. Please note that each follow
up response may take approximately 2 business days as the support
professional working with you may need further investigation to reach the
most efficient resolution. The offering is not appropriate for situations
that require urgent, real-time or phone-based interactions or complex
project analysis and dump analysis issues. Issues of this nature are best
handled working with a dedicated Microsoft Support Engineer by contacting
Microsoft Customer Support Services (CSS) at
http://msdn.microsoft.com/subscriptions/support/default.aspx.
======================================================
When responding to posts, please "Reply to Group" via
your newsreader so that others may learn and benefit
from this issue.
======================================================
This posting is provided "AS IS" with no warranties, and confers no rights.
======================================================