Tracking a memory leak.

[Frank Rizzo]

I have an object tree that is pretty gigantic and it holds about 100mb
of data. When I set the top object to null, I expect that the .NET
framework will clean up the memory at some point. However, I am looking
at the Task Manager and I don't see the MemUsage column decreasing even
after an hour or two. I know that TaskManager may not be the best place
to see what is the true way to gauge memory usage and/or presense of
memory leaks. So I am looking for a pointer on what I should use to see
whether setting an object to null really will at some point free up memory.

Regards

 

[Nicholas Paldino [.NET/C# MVP]]

Frank,

You should be looking at the performance counters here to see what the
memory consumption of .NET is. You should look at the ".NET CLR Memory"
category.

 

[Mr. Arnold]

I have an object tree that is pretty gigantic and it holds about 100mb of
data. When I set the top object to null, I expect that the .NET framework
will clean up the memory at some point. However, I am looking at the Task
Manager and I don't see the MemUsage column decreasing even after an hour
or two. I know that TaskManager may not be the best place to see what is
the true way to gauge memory usage and/or presense of memory leaks. So I
am looking for a pointer on what I should use to see whether setting an
object to null really will at some point free up memory.

Process Explorer may be of some help to you.

http://www.microsoft.com/technet/sysinternals/utilities/ProcessExplorer.mspx

You can right-click in the upper pane on a process and select Properties.

You can right-click in the lower-pane too.

 

[Bruce Wood]

I have an object tree that is pretty gigantic and it holds about 100mb
of data. When I set the top object to null, I expect that the .NET
framework will clean up the memory at some point. However, I am looking
at the Task Manager and I don't see the MemUsage column decreasing even
after an hour or two. I know that TaskManager may not be the best place
to see what is the true way to gauge memory usage and/or presense of
memory leaks. So I am looking for a pointer on what I should use to see
whether setting an object to null really will at some point free up memory.

Some reminders, and a question.

Remember that the garbage collector will reclaim memory only when you
put memory pressure on it. That is, only when you try to allocate more
memory and the GC detects that you're short. Also remember that if the
objects in your tree have been around for a while, they have likely
been promoted into higher generations of the GC, and are therefore
less likely to be reclaimed rapidly. The longer instances stay around,
the less likely they are to be reclaimed quickly.

Of course, if your program is _running_ for an hour or two, and keeps
allocating new objects during that time, and the contents of the
object tree _still_ don't go away, then you have a problem.

Now the question: does any of the objects in the tree subscribe to
events generated by objects outside the tree?

 

[Willy Denoyette [MVP]]

I have an object tree that is pretty gigantic and it holds about 100mb of
data. When I set the top object to null, I expect that the .NET framework
will clean up the memory at some point. However, I am looking at the Task
Manager and I don't see the MemUsage column decreasing even after an hour
or two. I know that TaskManager may not be the best place to see what is
the true way to gauge memory usage and/or presense of memory leaks. So I
am looking for a pointer on what I should use to see whether setting an
object to null really will at some point free up memory.

Regards

You should take a look at the "CLR Memory performance" counters, notably the
Gen0, 1, 2 and Large Object heap counters are of interest.
What is shown in Taskman is the size of your process Working Set or the
Private Bytes (depending the counter you are looking at), the GC allocates
and de-allocates object space from a Private Heap holding the above
generational heaps, this Private Heap,w which is part of the Private Bytes
and the Working Set, grows, by requesting new segments from the OS, whenever
the Generational heap 0 is full and the GC needs more space to allocate
objects from.
However, this heap will not necessarily decrease when objects are getting
GC'd, all the GC does is free the object space and compact the generational
heaps (0, or 0 and 1, or 0, 1 and 2).
The CLR "Memory Allocator" de-commits (after compactation) the pages that
were previously occupied by object data, and returns the Segment to the OS
when *all pages* from that segment (but not the default segments) are
decommited.
Note that pinned objects can spoil the party, as these cannot move to
another segment and as such prevents the Segment to become de-committed
completely. Another thing that you need to watch for is the LOH, as this
heap is never compacted, keeping objects alive in one of the possible extra
segment for this LOH, will prevent the segment to return to the OS.

Willy.

 

[Alvin Bruney [MVP]]

of data. When I set the top object to null, I expect that the .NET

framework will clean up the memory at some point. However, I am looking

Why do you expect this? A tree of n nodes where (n + n1 + n2 + n3...) with n
set to null is still linked at n1 + n2 + n3...). These are valid roots which
won't be garbage collected. I fully expect this to leak memory.

--
Regards,
Alvin Bruney
------------------------------------------------------
Shameless author plug
Excel Services for .NET is coming...
OWC Black book on Amazon and
www.lulu.com/owc
Professional VSTO 2005 - Wrox/Wiley

 

[Bruce Wood]

Why do you expect this? A tree of n nodes where (n + n1 + n2 + n3...) with n
set to null is still linked at n1 + n2 + n3...). These are valid roots which
won't be garbage collected. I fully expect this to leak memory.

Alvin, perhaps I don't understand your post.

If I have a tree of objects, with a single root pointer pointing to
the root node, and that pointing to n other nodes, which in turn point
to n other nodes, etc, and I set the root pointer to null, then entire
tree becomes available for collection, even though parts of itself
refer to other parts.

Whether the GC bothers to collect it, and when, is a whole other
question.

Why do you think that a structure like this would "lead memory"?

 

[Alvin Bruney [MVP]]

Yes, I see what you are saying. I don't know how trees work internally
either, however I suspect that they are linked lists of sorts. That would
mean that even though the root is null, the third node or n+1 still has a
bonafide root which is the node n. The second node does not have a root as
you correctly pointed out. Can this be collected?

I think it depends on which node is examined first. If the GC examines node
0, a collection for the entire tree is possible, however for any other node
examined first, these have valid roots which won't be collected.

The best way to resolve this is to write a short program that forces a
collection on a tree of length n and examine the heap - then we know for
sure. Any body got time on their hands?

--
Regards,
Alvin Bruney
------------------------------------------------------
Shameless author plug
Excel Services for .NET is coming...
OWC Black book on Amazon and
www.lulu.com/owc
Professional VSTO 2005 - Wrox/Wiley

 

[Nicholas Paldino [.NET/C# MVP]]

You don't need to write a short program.

If you have a tree where the nodes are only referenced by the parent,
and then the reference to the root is let go, then the whole tree is
eligible for collection.

Eligibility for GC doesn't matter if something is holding a reference to
an object or not, it's whether or not something is holding a reference to
that, and so on.

Basically, the CLR performs a mark and sweep, where it starts with the
stack on every thread (for the most part, there are other places it looks
which are not thread specific) and looks to see what references are held,
and then what references they hold, and so on, and so on. Once all of those
references are marked, anything else that is not marked is GC'ed.

So in this case, if you have a tree which has been let go, and nothing
is referencing it, or it's nodes, even though the parent references the
child nodes, the whole thing is eligible for GC.

 

[Ian Semmel]

I think that even if the top object is null, if something is referencing the
..Nodes, they will not be cleaned up, so the tree structure is still pretty
well intact.

 

[Frank Rizzo]

Now the question: does any of the objects in the tree subscribe to
events generated by objects outside the tree?

Bruce, the objects in the tree do not do subscribe or produce an events.

 

[Frank Rizzo]

You don't need to write a short program.

If you have a tree where the nodes are only referenced by the parent,
and then the reference to the root is let go, then the whole tree is
eligible for collection.

Eligibility for GC doesn't matter if something is holding a reference to
an object or not, it's whether or not something is holding a reference to
that, and so on.

Basically, the CLR performs a mark and sweep, where it starts with the
stack on every thread (for the most part, there are other places it looks
which are not thread specific) and looks to see what references are held,
and then what references they hold, and so on, and so on. Once all of those
references are marked, anything else that is not marked is GC'ed.

So in this case, if you have a tree which has been let go, and nothing
is referencing it, or it's nodes, even though the parent references the
child nodes, the whole thing is eligible for GC.

What if the child nodes reference its parent? Is it still eligible for
collection?

 

[Bruce Wood]

What if the child nodes reference its parent? Is it still eligible for
collection?

Yes. It doesn't matter if an object instance has references to other
(live) objects. All that matters is whether any live object has a
reference to the object instance being considered for collection.

An instance can have dozens of pointers to live objects and still be
collected, so long as no live object has a reference to IT. (Where
"live" means that it is referenced from the stack, from a static
variable, or one of a few other "root" places, or from another "live"
object.)

 

[Bruce Wood]

Bruce, the objects in the tree do not do subscribe or produce an events.

Hmm. Then I'm at a loss. Events are usually the type of reference that
everyone forgets about.

Can you post a stripped-down program that exhibits the behaviour? One
that we can all compile and run, and see it happen?

 

[Frank Rizzo]

Hmm. Then I'm at a loss. Events are usually the type of reference that
everyone forgets about.

Can you post a stripped-down program that exhibits the behaviour? One
that we can all compile and run, and see it happen?

Unfortunately not, the program is pretty complicated and has a
dependency on a 30gb database.

 

[Alvin Bruney [MVP]]

Nicholas:
Your assumption is that the initial sweep starts at the top most node in a
tree. There is nothing in the docs that I have found that indicates that
this is so. Perhaps you can point it out.

--
Regards,
Alvin Bruney
------------------------------------------------------
Shameless author plug
Excel Services for .NET is coming...
OWC Black book on Amazon and
www.lulu.com/owc
Professional VSTO 2005 - Wrox/Wiley

You don't need to write a short program.

If you have a tree where the nodes are only referenced by the parent,
and then the reference to the root is let go, then the whole tree is
eligible for collection.

Eligibility for GC doesn't matter if something is holding a reference
to an object or not, it's whether or not something is holding a reference
to that, and so on.

Basically, the CLR performs a mark and sweep, where it starts with the
stack on every thread (for the most part, there are other places it looks
which are not thread specific) and looks to see what references are held,
and then what references they hold, and so on, and so on. Once all of
those references are marked, anything else that is not marked is GC'ed.

So in this case, if you have a tree which has been let go, and nothing
is referencing it, or it's nodes, even though the parent references the
child nodes, the whole thing is eligible for GC.


--
- Nicholas Paldino [.NET/C# MVP]
- (e-mail address removed)

Yes, I see what you are saying. I don't know how trees work internally
either, however I suspect that they are linked lists of sorts. That would
mean that even though the root is null, the third node or n+1 still has a
bonafide root which is the node n. The second node does not have a root
as you correctly pointed out. Can this be collected?

I think it depends on which node is examined first. If the GC examines
node 0, a collection for the entire tree is possible, however for any
other node examined first, these have valid roots which won't be
collected.

The best way to resolve this is to write a short program that forces a
collection on a tree of length n and examine the heap - then we know for
sure. Any body got time on their hands?

--
Regards,
Alvin Bruney
------------------------------------------------------
Shameless author plug
Excel Services for .NET is coming...
OWC Black book on Amazon and
www.lulu.com/owc
Professional VSTO 2005 - Wrox/Wiley

 

[Jeffrey Tan[MSFT]]

Hi Frank,

Thanks for your feedback.

Actually, whether there is still any reference to an object is not the key
point for identifying an object as eligible for collecting. The GC finds
all the *live* objects through the *ROOT*. All the global and static object
pointers in an application are considered part of the application's roots.
In addition, any local variable/parameter object pointers on a thread's
stack are considered part of the application's roots. Finally, any CPU
registers containing pointers to objects in the managed heap are also
considered part of the application's roots. The list of active roots is
maintained by the just-in-time (JIT) compiler and common language runtime,
and is made accessible to the garbage collector's algorithm.

So in your scenario, although parent node references the child nodes and
child nodes reference the parent node, there is no root points to any
object in the tree, so the GC will not identify them as *live* object. So
they are eligible for collecting. Jeffrey Richter's famous article below
talks about GC algorithm in details:
"Garbage Collection: Automatic Memory Management in the Microsoft .NET
Framework"
http://msdn.microsoft.com/msdnmag/issues/1100/gci/

Hope it helps.

Best regards,
Jeffrey Tan
Microsoft Online Community Support
==================================================
Get notification to my posts through email? Please refer to
http://msdn.microsoft.com/subscriptions/managednewsgroups/default.aspx#notif
ications.

Note: The MSDN Managed Newsgroup support offering is for non-urgent issues
where an initial response from the community or a Microsoft Support
Engineer within 1 business day is acceptable. Please note that each follow
up response may take approximately 2 business days as the support
professional working with you may need further investigation to reach the
most efficient resolution. The offering is not appropriate for situations
that require urgent, real-time or phone-based interactions or complex
project analysis and dump analysis issues. Issues of this nature are best
handled working with a dedicated Microsoft Support Engineer by contacting
Microsoft Customer Support Services (CSS) at
http://msdn.microsoft.com/subscriptions/support/default.aspx.
==================================================
This posting is provided "AS IS" with no warranties, and confers no rights.

 

[Bill Butler]

Nicholas:
Your assumption is that the initial sweep starts at the top most node
in a tree. There is nothing in the docs that I have found that
indicates that this is so. Perhaps you can point it out.

If you re-read what he posted
you will find that he made no such assumption.
Unless there is a live chain of references that lead to these
nodes...They are eligible for collection

--
Regards,
Alvin Bruney
------------------------------------------------------
Shameless author plug
Excel Services for .NET is coming...
OWC Black book on Amazon and
www.lulu.com/owc
Professional VSTO 2005 - Wrox/Wiley

<snip>

 

[Bruce Wood]

No, he didn't make that assumption.

He correctly stated that the initial sweep doesn't start in the tree
at all. It starts on the _stack_ (and in other places mentioned by
Jeffrey Tan in his post). It then proceeds to follow all references
that fan out from that point. Any object not thus marked is eligible
for collection.

IF the root of the tree is referenced by a "live" object (that has
been found by starting from the stack, or a static object, or a
register, etc), then YES, the GC will follow the pointers from the
tree root to its child nodes, and their child nodes, and will traverse
the tree in some fashion, marking all of its nodes as ineligible for
collection.

If the only thing referenced by an object outside the tree is the
tree's root, AND you remove that reference, then no object in the tree
will be found during the mark phase, because nothing "live" points to
any part of the tree.

I think that the root of your misunderstanding (pardon the pun) is
that you seem to think that objects are garbage collected if they are
"marked for collection"... that is, you seem to be reasoning from the
angle that says that objects are collected if they are positively
marked. The opposite is true: objects are collected if they are _not
marked_ as "live". Mark-and-sweep finds everything that should _not_
be collected, then collects everything else.

 

[Alvin Bruney [MVP]]

I think that the root of your misunderstanding (pardon the pun) is
that you seem to think that objects are garbage collected if they are
"marked for collection"...

Not true, and I don't see how you got that from my argument either.

--
Regards,
Alvin Bruney
------------------------------------------------------
Shameless author plug
Excel Services for .NET is coming...
OWC Black book on Amazon and
www.lulu.com/owc
Professional VSTO 2005 - Wrox/Wiley