Large object heap and OutOfMemory Exception

[ermonnezzaro]

hello there,

I have developped a windows services in vb.net (framework 1.1 sp1).
I read PDF files stored in a database and put them in an arraylist to
create a unique document at the end of the loop.
Every single document is neary 40kb my arraylist can stored 150
elements maximum.
On a windows 2003 server I have problems (OutOfMemory) after few days
but this doesn't happen in WinXp.
I read nearly 200 post in different forums talking about "large object
heap" and the fact it's not compacted.
Most people suggest not to use these objects. Other gurus say you
should never call GC.Collect.
Well, nobody seem to have a real solution to avoid my application blow
up.
Is this .NET Framework so stupid when dealing with objects in memory?!
What do we have to do if we need to manage few files, in memory?
Where are the M$ guys when we need them?

Regards

Alberto

 

[Chris Mullins]

[The Large Object Heap]

On a windows 2003 server I have problems (OutOfMemory) after few days
but this doesn't happen in WinXp.

What makes you think the Large Object Heap is the culprit? With the .NET
Memory Performance Counters loaded, does the LOH show that it's growing
continually?

My experience is that it's much more common to have a Generation 2 memory
leak cause this problem than an LOH issue.

Are you running the Server or Workstation CLR? Does either of these machines
have more than one processor?

Most people suggest not to use these objects. Other gurus say you
should never call GC.Collect.

How do you think calling GC.Collect is going to help the issue?

Well, nobody seem to have a real solution to avoid my application blow
up.

Nobody should. It's your application. You haven't exactly helped "us" track
down the problem either at this point.

Is this .NET Framework so stupid when dealing with objects in memory?!

I would venture to say the .Net framework is the smartest object and memory
manger in production across any system or platform. There might be a few
things that give it a run for it's money, but it's gonna be close.

Where are the M$ guys when we need them?

The MS guys (or anyone who does this for a living) is going to tell you:
1) Look at the performance counters for member to determine where the
problem likley is.

2) Capture a Minidump when you get OOM, load it into WinDebug and Son of
Strike, and look at the heap.

This is all work you are capable of doing. I understand your frustration,
having been there myself, but by far the most likley canidate here is the
code that you wrote has a memory leak in it. It's not going to be a
framework bug. It's unlikley to be a LOH issue. It's very likley to be you
not understanding how Garbage Collection works yet, so you're leaking
memory.

If it does turn out to be the Large Object Heap, you're going to want to
switch to .Net 2.0, as it does perform compaction of the Large Object Heap.

 

[ermonnezzaro]

Hello Chris,

yes the server is a win2003 biprocessor and I have installed the
framework 1.1sp1
and I have my directive <gcServer enabled="true" /> in the application
cofiguration file.
I have made a profiling and it seems the Gen 2 heap size and are
growing progressively.
I have the problem after few days the windows services works.
With the GC.Collect I have solved my problem cause I can see the memory
is released
after the execution of the "hard work".
I don't know what's going to happen in 20 days, though.
I don't think the garbage collector is a good guy (I am talking about
framework 1.1)
cause I have read nearly 100 post in different newsgroups and many
people have the same
problems when dealing with datasets or binary objects.
I don't understand why on my WinXp everything works great and on a
server like that I do
have to worry about those stuff?
Are there any differences if I install my windows services in DEBUG or
RELEASE?

thank

regards
Alberto

 

[Chris Mullins]

[Memory Leaks]

yes the server is a win2003 biprocessor and I have installed the
framework 1.1sp1 and I have my directive <gcServer
enabled="true" /> in the application cofiguration file.

The first thing you want to do is turn that off. The gargage collection and
heap management algorithms are very different between the server and
workstation CLR, and that's only serving to confuse the issue at this point.

You want to use the same runtime on your XP workstation and your 2003 Server
so that you get the same behavior on both machines.

There are lots of resources on this available on the web. Here's a very
consice one...
http://blogs.msdn.com/clyon/archive/2004/09/08/226981.aspx

I have made a profiling and it seems the Gen 2 heap size and are
growing progressively.

This means that you're leaking memory. This isn't a garbage collection bug,
or a Large Object Heap bug, but a programming bug on your part. This is good
news, as it means you can fix the problem.

Go and download the SciTech Memory profiler and learn how to use it. Once
your comfortable with the tool, and have fixed the obvious low hanging
fruit, you're going to want to run your app for an hour, collect a heap
snapshot, let it run for a few more hours, and collect another heap
snapshot. Save this to a file somewhere, and spend the next 2 or 3 days
figuring out what's being leaked.

It will take you a full day or two just to learn how to interpret the data
given to you by the tool. This first thing you'll want to do is make sure
Dispose Tracking is turned on, get the report of "Should have been disposed
but wasn't" and fix these - hopefully fixing tje, will probably be pretty
quick. Then Lather, Rinse, Repeat.

With the GC.Collect I have solved my problem cause I can see the memory
is released after the execution of the "hard work".

It's really not. It's looking like fix, but it really hasn't solved the
problem at all. Trust me, I've been down this path before.

I don't think the garbage collector is a good guy (I am talking about
framework 1.1) cause I have read nearly 100 post in different
newsgroups and many people have the same problems when dealing
with datasets or binary objects.

Yea, well, don't believe everything what you read on the internet. People
become frustrated and confused when lacking the GC Gnosis.... and often find
a good scapegoat in the Garbage Collector. They're no more right than the
Greeks who believed Thunder was caused by the gods.

The Garbage Collector isn't the problem here. It doesn't have a bug. You
have legit memory leaks in your app.

I don't understand why on my WinXp everything works great and on a
server like that I do have to worry about those stuff?

Your problem is that you read a post somewhere that said to use the line:
<gcServer enabled="true" />
so that your application will run better. You did this without taking the
time to understand what this means - and it has changed the way your
application behaves in such a way at to expose your memory management bugs
much more quickly.

You're using the Server CLR on the server, and the workstation CLR on your
workstation. They are very different. You can go read all about it on MSDN.

There's nothing wrong with the server CLR GC Algorithms - they're not buggy
either. It's just that the way they work makes your problem surface more
quickly, as they split the manged heaps up across processors.

Are there any differences if I install my windows services in DEBUG or
RELEASE?

At this point in time? Not really. You need to fix the memory leaks, and
they're going to exist in both Debug and Release mode.

 

[ermonnezzaro]

Hello Chris,

thank you for your reply.
You're right I should understand more about the garbage collector
mechanism.
Now I am going to try to use these profilers to see what's going on in
the application
even if I could say my application is nothing special.
This services fetch binary data on a database, create a collection of
binaries and build a PDF document
with all these stuff. I am using iTextSharp to generate the PDF file.
I think I am going to debug that component as well.
Anyway, I would like to say that I don't read whatever I get on the
internet and think it's true
but I try to understand.
I found the <gcServer enabled="true" /> directive cause I found a
problem with the application.
I discovered it and studied the implication of its use. At the end I
realized nothing had changed.

regards

Alberto