Memory Leak in .NET 1.1

[MAG1301]

I've detected memory leaks in our huge .NET 1.1 C# application but
couldn't localize them directly. So I've reduced the code to the
following console application:

using System;
using System.IO;

namespace MemLeak
{
class MemLeak
{
[STAThread]
static void Main(string[] args)
{
try
{
throw new NullReferenceException();
}
catch
{
}

for(int i=0; i<1000; i++)
{
using(StreamWriter newMasterfile =
new StreamWriter(@"c:\XXX.tmp"))
{
newMasterfile.Close();
}
}

Console.ReadLine();
}
}
}

Also you have to create an "App.config" file which is distributed by
VS2003 into the application directory with the name of the application
followed by ".config". There may be any content, the file can be empty
too - this will not influence our result.

Compile the "Release" configuration and you'll see with a tool like
"NetMemoryProfiler" that there are almost 1000 StreamWriter reserved at
"Console.ReadLine". You can wait as long as you want - they'll never be
freed. If you don't have such a tool: increase the number of loops.
Then you can use the task manager to see how the memory grows (look at
the virtual memory!). With an endless loop there will be an "Out of
virtual memory" error at the end.

This problem is not only bounded to StreamWriter. We've seen this for
database objects, simple strings and other.

We have tested this with several Windows XP SP2 Systems with different
configurations. Also with the latest fixes for Windows and .NET.

The funny thing: there will be no leaks when you use the "Debug"
configuration. Also you can omit the exception or the "App.config"
file. In these cases the memory will be freed correctly. But we will
comile in "Release" configuration, use the "App.config" and Exceptions
can't be avoided.

The last option is to switch the attribute "[STAThread]" to
"[MTAThread]". The memory is ok in the small example (why?) and also in
our huge application. But I'm not sure with the COM-components we use.
So I prefer the "[STAThread]" attribute.

There are some methods which you can tell the gabage collector to clean
up: "GC.GetTotalMemory(...)", "GC.WaitForPendingFinalizers()" and
"GC.Collect()". You can find combinations in the small example to get
rid of the memory. But this is very unreliable and will not work
correctly in our huge application where late binding and remoting is
used.

After some days of trial and error, I am really at my wits' end over
this problem. So is there anybody who can help me?

Many thanks in advance
Martin Gaus

 

[Ludwig]

I've detected memory leaks in our huge .NET 1.1 C# application but
couldn't localize them directly. So I've reduced the code to the
following console application:

using System;
using System.IO;

namespace MemLeak
{
class MemLeak
{
[STAThread]
static void Main(string[] args)
{
try
{
throw new NullReferenceException();
}
catch
{
}

for(int i=0; i<1000; i++)
{
using(StreamWriter newMasterfile =
new StreamWriter(@"c:\XXX.tmp"))
{
newMasterfile.Close();
}
}

Console.ReadLine();
}
}
}

Also you have to create an "App.config" file which is distributed by
VS2003 into the application directory with the name of the application
followed by ".config". There may be any content, the file can be empty
too - this will not influence our result.

Compile the "Release" configuration and you'll see with a tool like
"NetMemoryProfiler" that there are almost 1000 StreamWriter reserved at
"Console.ReadLine". You can wait as long as you want - they'll never be
freed. If you don't have such a tool: increase the number of loops.
Then you can use the task manager to see how the memory grows (look at
the virtual memory!). With an endless loop there will be an "Out of
virtual memory" error at the end.

This problem is not only bounded to StreamWriter. We've seen this for
database objects, simple strings and other.

We have tested this with several Windows XP SP2 Systems with different
configurations. Also with the latest fixes for Windows and .NET.

The funny thing: there will be no leaks when you use the "Debug"
configuration. Also you can omit the exception or the "App.config"
file. In these cases the memory will be freed correctly. But we will
comile in "Release" configuration, use the "App.config" and Exceptions
can't be avoided.

The last option is to switch the attribute "[STAThread]" to
"[MTAThread]". The memory is ok in the small example (why?) and also in
our huge application. But I'm not sure with the COM-components we use.
So I prefer the "[STAThread]" attribute.

There are some methods which you can tell the gabage collector to clean
up: "GC.GetTotalMemory(...)", "GC.WaitForPendingFinalizers()" and
"GC.Collect()". You can find combinations in the small example to get
rid of the memory. But this is very unreliable and will not work
correctly in our huge application where late binding and remoting is
used.

After some days of trial and error, I am really at my wits' end over
this problem. So is there anybody who can help me?

Many thanks in advance
Martin Gaus

just wondering (without trying it)

newMasterfile.Close();
newMasterfile=null;

does this make any difference?

 

[Martin]

using(StreamWriter newMasterfile = new StreamWriter( @"c:\XXX.tmp" ))
{
newMasterfile.Close();
newMasterfile = null;
}

Something like this is not possible. The compiler complains that
"newMasterfile" is write protected. But I've tested the following with
the same result (with leaks):

StreamWriter newMasterfile = new StreamWriter( @"c:\XXX.tmp" );
newMasterfile.Close();
newMasterfile = null;

 

[Greg Young]

There does not seem to be a memory leak here (atleast on my system) .. try
the included code .. on my system it gets up to about 8mb of memory .. then
flattens out (if there were a memory leak here I can assure you that running
10,000,000 times as opposed to 1000 would show a severe difference in memory
usage)

Most likely the garbage collector has just not collected those items yet
(and with no new items being created, and no pressure on the memory it
already has allocated why should it hurry?). Once some pressure is placed
upon it the gc seems to pick things up without issue

In your larger system my first place to look given the things you mention
would be the large object heap to see if it is growing .. if it is, whats in
it?

Cheers,

Greg Young
MVP - C#

using System;
using System.IO;

namespace MemLeak
{
class MemLeak
{
[STAThread]
static void Main(string[] args)
{
try
{
throw new NullReferenceException();
}
catch
{
}

for(int i=0; i<10000000; i++)
{
using(StreamWriter newMasterfile =
new StreamWriter(@"c:\XXX.tmp"))
{
newMasterfile.Close();
}
}

Console.ReadLine();
}
}
}

I've detected memory leaks in our huge .NET 1.1 C# application but
couldn't localize them directly. So I've reduced the code to the
following console application:

using System;
using System.IO;

namespace MemLeak
{
class MemLeak
{
[STAThread]
static void Main(string[] args)
{
try
{
throw new NullReferenceException();
}
catch
{
}

for(int i=0; i<1000; i++)
{
using(StreamWriter newMasterfile =
new StreamWriter(@"c:\XXX.tmp"))
{
newMasterfile.Close();
}
}

Console.ReadLine();
}
}
}

Also you have to create an "App.config" file which is distributed by
VS2003 into the application directory with the name of the application
followed by ".config". There may be any content, the file can be empty
too - this will not influence our result.

Compile the "Release" configuration and you'll see with a tool like
"NetMemoryProfiler" that there are almost 1000 StreamWriter reserved at
"Console.ReadLine". You can wait as long as you want - they'll never be
freed. If you don't have such a tool: increase the number of loops.
Then you can use the task manager to see how the memory grows (look at
the virtual memory!). With an endless loop there will be an "Out of
virtual memory" error at the end.

This problem is not only bounded to StreamWriter. We've seen this for
database objects, simple strings and other.

We have tested this with several Windows XP SP2 Systems with different
configurations. Also with the latest fixes for Windows and .NET.

The funny thing: there will be no leaks when you use the "Debug"
configuration. Also you can omit the exception or the "App.config"
file. In these cases the memory will be freed correctly. But we will
comile in "Release" configuration, use the "App.config" and Exceptions
can't be avoided.

The last option is to switch the attribute "[STAThread]" to
"[MTAThread]". The memory is ok in the small example (why?) and also in
our huge application. But I'm not sure with the COM-components we use.
So I prefer the "[STAThread]" attribute.

There are some methods which you can tell the gabage collector to clean
up: "GC.GetTotalMemory(...)", "GC.WaitForPendingFinalizers()" and
"GC.Collect()". You can find combinations in the small example to get
rid of the memory. But this is very unreliable and will not work
correctly in our huge application where late binding and remoting is
used.

After some days of trial and error, I am really at my wits' end over
this problem. So is there anybody who can help me?

Many thanks in advance
Martin Gaus

 

[Ludwig]

using(StreamWriter newMasterfile = new StreamWriter( @"c:\XXX.tmp" ))
{
newMasterfile.Close();
newMasterfile = null;
}

Something like this is not possible. The compiler complains that
"newMasterfile" is write protected. But I've tested the following with
the same result (with leaks):

StreamWriter newMasterfile = new StreamWriter( @"c:\XXX.tmp" );
newMasterfile.Close();
newMasterfile = null;

Well I tried it and used the performance monitor to monitor the
available bytes. I used a loop of 1000000000.
Result: number of available bytes decreased slowly. After 10 minutes
it went up again. After 20 minutes I see no evidence of a memory leak.

 

[Nicholas Paldino [.NET/C# MVP]]

With this example, why ^wouldn't^ there be a bunch of StreamWriter
objects? The way that GC works, unless there was a prior GC, those objects
will reside in memory. Just because you close them doesn't mean that the
instances are still on the heap. Only the resources that they manage which
are critical (in this case, the file handle for the underlying stream) are
released when Close is called.

Then, you call ReadLine, which uses P/Invoke ultimately. Since it is
waiting in an unmanaged piece of code (and more than likely, you are running
the CLR in a workstation configuration with the GC occuring on the executing
thread), you aren't going to be able to have a GC occur.

What you are seeing in this example is completely predictable and
expected.


--
- Nicholas Paldino [.NET/C# MVP]
- (e-mail address removed)

I've detected memory leaks in our huge .NET 1.1 C# application but
couldn't localize them directly. So I've reduced the code to the
following console application:

using System;
using System.IO;

namespace MemLeak
{
class MemLeak
{
[STAThread]
static void Main(string[] args)
{
try
{
throw new NullReferenceException();
}
catch
{
}

for(int i=0; i<1000; i++)
{
using(StreamWriter newMasterfile =
new StreamWriter(@"c:\XXX.tmp"))
{
newMasterfile.Close();
}
}

Console.ReadLine();
}
}
}

Also you have to create an "App.config" file which is distributed by
VS2003 into the application directory with the name of the application
followed by ".config". There may be any content, the file can be empty
too - this will not influence our result.

Compile the "Release" configuration and you'll see with a tool like
"NetMemoryProfiler" that there are almost 1000 StreamWriter reserved at
"Console.ReadLine". You can wait as long as you want - they'll never be
freed. If you don't have such a tool: increase the number of loops.
Then you can use the task manager to see how the memory grows (look at
the virtual memory!). With an endless loop there will be an "Out of
virtual memory" error at the end.

This problem is not only bounded to StreamWriter. We've seen this for
database objects, simple strings and other.

We have tested this with several Windows XP SP2 Systems with different
configurations. Also with the latest fixes for Windows and .NET.

The funny thing: there will be no leaks when you use the "Debug"
configuration. Also you can omit the exception or the "App.config"
file. In these cases the memory will be freed correctly. But we will
comile in "Release" configuration, use the "App.config" and Exceptions
can't be avoided.

The last option is to switch the attribute "[STAThread]" to
"[MTAThread]". The memory is ok in the small example (why?) and also in
our huge application. But I'm not sure with the COM-components we use.
So I prefer the "[STAThread]" attribute.

There are some methods which you can tell the gabage collector to clean
up: "GC.GetTotalMemory(...)", "GC.WaitForPendingFinalizers()" and
"GC.Collect()". You can find combinations in the small example to get
rid of the memory. But this is very unreliable and will not work
correctly in our huge application where late binding and remoting is
used.

After some days of trial and error, I am really at my wits' end over
this problem. So is there anybody who can help me?

Many thanks in advance
Martin Gaus

 

[Martin]

I'm wondering why you can't approve my observations.
I've seen this on several installations now.

Just to be sure:
- it must be the "Release" configuration
- in directory "bin\Release" there must be an application cofiguration
file.
I have two files there:
MemoryLeak.exe
MemoryLeak.exe.config

When I start my tests the virtual memory of the process will grow in
one minute from 5MB to 17MB and so on until all the virtual memory is
completly used. There will never be a gabage collection neither when
the memory is under pressure.

May I send you my solution incl. binary?

Martin

 

[Lebesgue]

And how would you explain that the problem goes away when they delete the
app.config file from the bin directory?

With this example, why ^wouldn't^ there be a bunch of StreamWriter
objects? The way that GC works, unless there was a prior GC, those
objects will reside in memory. Just because you close them doesn't mean
that the instances are still on the heap. Only the resources that they
manage which are critical (in this case, the file handle for the
underlying stream) are released when Close is called.

Then, you call ReadLine, which uses P/Invoke ultimately. Since it is
waiting in an unmanaged piece of code (and more than likely, you are
running the CLR in a workstation configuration with the GC occuring on the
executing thread), you aren't going to be able to have a GC occur.

What you are seeing in this example is completely predictable and
expected.


--
- Nicholas Paldino [.NET/C# MVP]
- (e-mail address removed)

I've detected memory leaks in our huge .NET 1.1 C# application but
couldn't localize them directly. So I've reduced the code to the
following console application:

using System;
using System.IO;

namespace MemLeak
{
class MemLeak
{
[STAThread]
static void Main(string[] args)
{
try
{
throw new NullReferenceException();
}
catch
{
}

for(int i=0; i<1000; i++)
{
using(StreamWriter newMasterfile =
new StreamWriter(@"c:\XXX.tmp"))
{
newMasterfile.Close();
}
}

Console.ReadLine();
}
}
}

Also you have to create an "App.config" file which is distributed by
VS2003 into the application directory with the name of the application
followed by ".config". There may be any content, the file can be empty
too - this will not influence our result.

Compile the "Release" configuration and you'll see with a tool like
"NetMemoryProfiler" that there are almost 1000 StreamWriter reserved at
"Console.ReadLine". You can wait as long as you want - they'll never be
freed. If you don't have such a tool: increase the number of loops.
Then you can use the task manager to see how the memory grows (look at
the virtual memory!). With an endless loop there will be an "Out of
virtual memory" error at the end.

This problem is not only bounded to StreamWriter. We've seen this for
database objects, simple strings and other.

We have tested this with several Windows XP SP2 Systems with different
configurations. Also with the latest fixes for Windows and .NET.

The funny thing: there will be no leaks when you use the "Debug"
configuration. Also you can omit the exception or the "App.config"
file. In these cases the memory will be freed correctly. But we will
comile in "Release" configuration, use the "App.config" and Exceptions
can't be avoided.

The last option is to switch the attribute "[STAThread]" to
"[MTAThread]". The memory is ok in the small example (why?) and also in
our huge application. But I'm not sure with the COM-components we use.
So I prefer the "[STAThread]" attribute.

There are some methods which you can tell the gabage collector to clean
up: "GC.GetTotalMemory(...)", "GC.WaitForPendingFinalizers()" and
"GC.Collect()". You can find combinations in the small example to get
rid of the memory. But this is very unreliable and will not work
correctly in our huge application where late binding and remoting is
used.

After some days of trial and error, I am really at my wits' end over
this problem. So is there anybody who can help me?

Many thanks in advance
Martin Gaus

 

[Artur-ek]

There does not seem to be a memory leak here (atleast on my system)

the same here, no leak

System: Win 2003
Framework: 1.1.4322.2300
compiled in release mode, with config file in directory.

 

[Willy Denoyette [MVP]]

| I've detected memory leaks in our huge .NET 1.1 C# application but
| couldn't localize them directly. So I've reduced the code to the
| following console application:
|
| using System;
| using System.IO;
|
| namespace MemLeak
| {
| class MemLeak
| {
| [STAThread]
| static void Main(string[] args)
| {
| try
| {
| throw new NullReferenceException();
| }
| catch
| {
| }
|
| for(int i=0; i<1000; i++)
| {
| using(StreamWriter newMasterfile =
| new StreamWriter(@"c:\XXX.tmp"))
| {
| newMasterfile.Close();
| }
| }
|
| Console.ReadLine();
| }
| }
| }
|
| Also you have to create an "App.config" file which is distributed by
| VS2003 into the application directory with the name of the application
| followed by ".config". There may be any content, the file can be empty
| too - this will not influence our result.
|
| Compile the "Release" configuration and you'll see with a tool like
| "NetMemoryProfiler" that there are almost 1000 StreamWriter reserved at
| "Console.ReadLine". You can wait as long as you want - they'll never be
| freed. If you don't have such a tool: increase the number of loops.
| Then you can use the task manager to see how the memory grows (look at
| the virtual memory!). With an endless loop there will be an "Out of
| virtual memory" error at the end.
|
| This problem is not only bounded to StreamWriter. We've seen this for
| database objects, simple strings and other.
|
| We have tested this with several Windows XP SP2 Systems with different
| configurations. Also with the latest fixes for Windows and .NET.
|
| The funny thing: there will be no leaks when you use the "Debug"
| configuration. Also you can omit the exception or the "App.config"
| file. In these cases the memory will be freed correctly. But we will
| comile in "Release" configuration, use the "App.config" and Exceptions
| can't be avoided.
|
| The last option is to switch the attribute "[STAThread]" to
| "[MTAThread]". The memory is ok in the small example (why?) and also in
| our huge application. But I'm not sure with the COM-components we use.
| So I prefer the "[STAThread]" attribute.
|
| There are some methods which you can tell the gabage collector to clean
| up: "GC.GetTotalMemory(...)", "GC.WaitForPendingFinalizers()" and
| "GC.Collect()". You can find combinations in the small example to get
| rid of the memory. But this is very unreliable and will not work
| correctly in our huge application where late binding and remoting is
| used.
|
| After some days of trial and error, I am really at my wits' end over
| this problem. So is there anybody who can help me?
|
| Many thanks in advance
| Martin Gaus
|

The small sample you posted does not leak, nor using V1.1.4322 nor using
v2.0., with or without a config file.
Your conclusion stems from the fact that you don't really understand how the
GC works, let met explain:

1. The only point in time a GC can occur is when objects are being created
on the GC heap, when waiting on a ReadLine(), no more objects are getting
instantiated, so, no GC can ever occur as long as you are waiting for a
keybord input.
That means that the StreamWriter instances who are not yet collected at the
moment you reach ReadLine, won't be cleaned-up before the process terminates
(well, the next GC).
Now, if you simply run this program from the console and watch the 'CLR
Memory' performance counters, you will see a number of Gen0 collections
occurring while executing the loop. That means that a bunch of StreamWriter
objects will be removed from the heap, while a number of StreamWrtiter
objects are still on the heap when executing ReadLine and as I said before
they stay there until the next GC (here because of the AD unload).
If you run this with a profiler attached, the behavior changes dramatically,
the CLR switches to a less aggressive mode of GC when a
(managed)debugger/profiler is attached, that means that it's possible that
no Gen0 collections occurred during the loop, leaving all StreamWriter
objects on the heap at the moment you reach ReadLine().
Ok, why the different behavior when there is a config file? Well, the
allocation/de-allocation scheme changed because you created a lot more
objects before you execute the loop, possibly triggering a GC run before the
loop has started and as such preventing a GC run during the loop.
Lesson learned, don't use the CLR memory profiler to monior such simple
programs, start using the non-invasive tools like perfmon to monitor your
memory allocation pattern before you start profiling using any managed
profilers, and if you do, make sure you understand what you are looking at.

2. you need to set the apartment state of the main thread to be compatible
with your COM objects requirements, seems like you don't know what they are.
When you set this to STA, be sure you never block this thread or you will
prevent the finalizer to clean-up the finalizable objects, possibly creating
a huge memory leak.

Willy.

 

[Jon Skeet [C# MVP]]

1. The only point in time a GC can occur is when objects are being created
on the GC heap, when waiting on a ReadLine(), no more objects are getting
instantiated, so, no GC can ever occur as long as you are waiting for a
keybord input.

That's not quite true, IMO. I believe GC will kick in when there's
memory pressure, too. Here are two programs which demonstrate it
between them:

Test.cs:
using System;

class Test
{
~Test()
{
Console.WriteLine ("Finalizer");
}

static void Main()
{
Test t = new Test();
Console.WriteLine ("Created");
Console.ReadLine();
}
}


UseMemory.cs:
using System.Collections;

class UseMemory
{
static void Main()
{
ArrayList list = new ArrayList();
while (true)
{
list.Add (new byte[10000]);
}
}
}

From one console, compile and run Test.cs. You'll see (or may see)
"Created" on its own. You can then leave that console alone for a
couple of minutes, and nothing is printed.

Then compile and run UseMemory.cs. Shortly, you should see "Finalizer"
being printed in the original console window - at least, that's what
happens on my box.

While this isn't absolute proof (due to finalizers and GC being
different things) it's at least extremely suggestive.

 

[Willy Denoyette [MVP]]

True, when there is physical memory pressure (high memory threshold) and you
are running on XP or higher, the OS will notify the CLR who will start to
trim it's working set, but before doing this, the CLR will trigger a full
collection, that means it will trigger the GC at the next allocation request
or whenever the program enters the CLR (or returns from unmanaged), but in
your case (unless you run a server GC), the GC cannot run as it's main
thread blocked in unmanaged code (result of the ReadLine()), so the CLR
proceeds by starting the finalizer thread (running on it's own thread), this
is exactly what you noticed when running your code.

I didn't want to complicate things any further by introducing "external
triggers", so I didn't mention these, I didn't count on you to come back on
this, I should have known better ;-).

Willy.



|
| <snip>
|
| > 1. The only point in time a GC can occur is when objects are being
created
| > on the GC heap, when waiting on a ReadLine(), no more objects are
getting
| > instantiated, so, no GC can ever occur as long as you are waiting for a
| > keybord input.
|
| That's not quite true, IMO. I believe GC will kick in when there's
| memory pressure, too. Here are two programs which demonstrate it
| between them:
|
| Test.cs:
| using System;
|
| class Test
| {
| ~Test()
| {
| Console.WriteLine ("Finalizer");
| }
|
| static void Main()
| {
| Test t = new Test();
| Console.WriteLine ("Created");
| Console.ReadLine();
| }
| }
|
|
| UseMemory.cs:
| using System.Collections;
|
| class UseMemory
| {
| static void Main()
| {
| ArrayList list = new ArrayList();
| while (true)
| {
| list.Add (new byte[10000]);
| }
| }
| }
|
| From one console, compile and run Test.cs. You'll see (or may see)
| "Created" on its own. You can then leave that console alone for a
| couple of minutes, and nothing is printed.
|
| Then compile and run UseMemory.cs. Shortly, you should see "Finalizer"
| being printed in the original console window - at least, that's what
| happens on my box.
|
| While this isn't absolute proof (due to finalizers and GC being
| different things) it's at least extremely suggestive.
|
| --
| Jon Skeet - <[email protected]>
| http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
| If replying to the group, please do not mail me too

 

[Martin]

Thats all fine. But why do I ran out of virtual memory when I have an
endless loop?

....
while(true)
{
using(StreamWriter newMasterfile =
new StreamWriter(@"c:\XXX.tmp"))
{
newMasterfile.Close();
}
}
....

 

[Jon Skeet [C# MVP]]

Thats all fine. But why do I ran out of virtual memory when I have an
endless loop?

...
while(true)
{
using(StreamWriter newMasterfile =
new StreamWriter(@"c:\XXX.tmp"))
{
newMasterfile.Close();
}
}

That I don't know - I can't reproduce it. How quickly do you leak
memory? Should I be able to easily see it in task manager?

 

[Martin]

I think so.
About 10MB per minute.
Intel P4 mobile, 2 GHz.

 

[Willy Denoyette [MVP]]

| Thats all fine. But why do I ran out of virtual memory when I have an
| endless loop?
|
| ...
| while(true)
| {
| using(StreamWriter newMasterfile =
| new StreamWriter(@"c:\XXX.tmp"))
| {
| newMasterfile.Close();
| }
| }
| ...
|

Don't know, it doesn't happen when I run your code, are you sure you are
running the exact same code as you posted?

Willy.

 

[Greg Young]

Why post you exact solution and we can check it out? The code posted at
present when brought here (with or without an app.config does not seem to
have any problems)

Cheers,

Greg Young
MVP - C#

 

[Martin]

Please give me a place where I can upload a zip file with the solution.

 

[Greg Young]

There are lots of free services such as http://www.savefile.com/ which offer
this exact functionality.

Cheers,

Greg Young

 

[Martin]

Good morning and thank you Greg!

you'll find the files there:
http://savefile.com/projects/831693

There are 2 files:
- MemoryLeak1: File with the original problem as described
- MemoryLeak2: A threading test with free main thread

Just start the compiled release.
Is there anyone who can reproduce my problem?

Martin