Finalizer Queue

[ajbecker]

I'm trying to debug a problem I'm having where my finalizer queue is
getting filled up faster than the thread can execute them, (or at least
I think that's what's going on.) Is there a way that I can see how
large the queue is or, even better, what objects are in the queue to be
finalized?

 

[Nicholas Paldino [.NET/C# MVP]]

There should be a performance counter that handles this.

However, what makes you think that this is the situation you are running
into? Is an exception being thrown, or something of that nature? What
exactly is going on?

 

[ajbecker]

That's a very good question. I wasn't very specific. Basically what's
going on is I have several threads running and each of them are
invoking certain managed functions, and those managed functions will be
using a variety of COM objects. The problem is, the faster I throw
items at the program to handle them (causing the managed functions to
get invoked.) The more the memory goes up. After a while (20 minutes
to an hour) the memory will be re-claimed. I've tracked the memory
increase to be that the COM objects are not getting released. I am
able to call ReleaseComObject on all the allocated COM objects and I
don't get the memory bloat. I've also tried wrapping the invoke in an
AppDomain, and that works too. There are several reasons that I don't
want to get into for why I can't use ReleaseComObject as a permanent
solution, and wrapping the invoke in an AppDomain causes a 25%
performance decrease which is unacceptable.

I came up with a theory that the COM objects were not getting released
because the finalizer thread was not getting around to calling the
finalizer on the RCW (Runtime Callable Wrapper) which would reduce the
reference count down to zero, thereby triggering the release of the COM
object. I searched through the code and found some finalizers that
could be taking longer than they should and manually released each of
those objects and suppressed their finallizers. By doing that, I was
able to decrease the memory bloat dramatically, however it still
occurs, and as I increase the load more, the memory bloat eventually
catches up to where it was before hand. From that experiment, and a
few others, I deduced that the COM objects were not released because
the finalizer thread was not getting around to them. Next I wanted to
figure out why.

Since I first posted this message, I did some more thinking and it
occurred to me that when the finalizer is called on the RCW, it seems
logical that the COM object will be released in that same thread. As a
result, the RCW itself may be what is bogging down the finallizer
thread.

I would also like to point out that all of this is, of course, theory,
and I could be completely wrong about what's going on. I'm currently
working on a work around to the problem, but I would much rather have a
more permanent solution

 

[Willy Denoyette [MVP]]

What are the threading requirements of your COM objects and what's the
apartment state of the thread holding a reference to their instances. The
reason I'm asking is that you'll have to pump the message queue if the
threads are in an STA, failing to pump will block the finalizer when he
tries to run your finalize method.
Willy.



| That's a very good question. I wasn't very specific. Basically what's
| going on is I have several threads running and each of them are
| invoking certain managed functions, and those managed functions will be
| using a variety of COM objects. The problem is, the faster I throw
| items at the program to handle them (causing the managed functions to
| get invoked.) The more the memory goes up. After a while (20 minutes
| to an hour) the memory will be re-claimed. I've tracked the memory
| increase to be that the COM objects are not getting released. I am
| able to call ReleaseComObject on all the allocated COM objects and I
| don't get the memory bloat. I've also tried wrapping the invoke in an
| AppDomain, and that works too. There are several reasons that I don't
| want to get into for why I can't use ReleaseComObject as a permanent
| solution, and wrapping the invoke in an AppDomain causes a 25%
| performance decrease which is unacceptable.
|
| I came up with a theory that the COM objects were not getting released
| because the finalizer thread was not getting around to calling the
| finalizer on the RCW (Runtime Callable Wrapper) which would reduce the
| reference count down to zero, thereby triggering the release of the COM
| object. I searched through the code and found some finalizers that
| could be taking longer than they should and manually released each of
| those objects and suppressed their finallizers. By doing that, I was
| able to decrease the memory bloat dramatically, however it still
| occurs, and as I increase the load more, the memory bloat eventually
| catches up to where it was before hand. From that experiment, and a
| few others, I deduced that the COM objects were not released because
| the finalizer thread was not getting around to them. Next I wanted to
| figure out why.
|
| Since I first posted this message, I did some more thinking and it
| occurred to me that when the finalizer is called on the RCW, it seems
| logical that the COM object will be released in that same thread. As a
| result, the RCW itself may be what is bogging down the finallizer
| thread.
|
| I would also like to point out that all of this is, of course, theory,
| and I could be completely wrong about what's going on. I'm currently
| working on a work around to the problem, but I would much rather have a
| more permanent solution
|

 

[Nicholas Paldino [.NET/C# MVP]]

I am able to call ReleaseComObject on all the allocated COM objects and I
don't get the memory bloat. I've also tried wrapping the invoke in an
AppDomain, and that works too. There are several reasons that I don't
want to get into for why I can't use ReleaseComObject as a permanent
solution, and wrapping the invoke in an AppDomain causes a 25%
performance decrease which is unacceptable.

I wouldn't expect you to run these objects in another app domain (unless
your design calls for it on some level), but I do think you have to explain
why you aren't using ReleaseComObject.

If you are not using it, it implies (and without knowing anything about
your app, I can't say for sure) that you don't have a grasp on your design
when it comes to object lifetimes. This is something that is critical for
COM object use.

If you could, can you explain why you can't or are not using it?

 

[ajbecker]

I'll have to look a little more into it. I thought all my COM objects
were MTA, but what your saying makes sence, and it sounds exactly like
what's going on.

 

[ajbecker]

Basicaly, I cannot call the ReleaseComObjects in the invoked method
because I cannot depend on it always being done that way. It would be
sort of like if an end user was writing that function and I couldn't
depend on them freeing up the memory properly.

 

[Willy Denoyette [MVP]]

| I'll have to look a little more into it. I thought all my COM objects
| were MTA, but what your saying makes sence, and it sounds exactly like
| what's going on.
|

COM objects can't be 'MTA', but I guess you mean 'Freethreaded'. Note that
objects that are marked 'Both', will end in the STA when that thread run's
in an STA, so this thread must pump, while they'll end in the sole MTA when
the thread runs in the MTA. Take also care, one single COM object running in
an STA thread that doesn't pump will deadlock the finalizer thread when
failing to pump (when relying on the finalizer to release the instance). So,
you may have thousands of MTA instances waiting to be finalized, because
there's one single STA object blocking the finalizer.

Willy.

 

[ajbecker]

I appreciate all the comments. Now I have a question that feels very
ignorant. How do I pump the thread(s)? In my app, the main
application is unmanaged.

 

[ajbecker]

hmm... this page [] says I can use
System.Threading.Thread.CurrentThread.Join(0), but that will only let
one message through at a time.. and what if there are several pending
messages?

 

[ajbecker]

yeah, that was smart, I didn't paste the URL in:
http://markorangel.com/blog/archive/2005/04/28/164.aspx

 

[Willy Denoyette [MVP]]

| hmm... this page [] says I can use
| System.Threading.Thread.CurrentThread.Join(0), but that will only let
| one message through at a time.. and what if there are several pending
| messages?
|

There are a number of pumping waits in the framework, Join(0) is one of
them.
Note that these "pumping waits" only pump "COM" messages, and while it's
true that only one message is pumped at a time, the Join(0) keeps pumping as
it will never stop as long as the thread exists.

Willy.

 

[Willy Denoyette [MVP]]

|
| || hmm... this page [] says I can use
|| System.Threading.Thread.CurrentThread.Join(0), but that will only let
|| one message through at a time.. and what if there are several pending
|| messages?
||
|
| There are a number of pumping waits in the framework, Join(0) is one of
| them.
| Note that these "pumping waits" only pump "COM" messages, and while it's
| true that only one message is pumped at a time, the Join(0) keeps pumping
as
| it will never stop as long as the thread exists.
|
| Willy.
|
|
Sorry, I meant to say .... a Join() keeps pumping....

Willy.

 

[Willy Denoyette [MVP]]

|I appreciate all the comments. Now I have a question that feels very
| ignorant. How do I pump the thread(s)? In my app, the main
| application is unmanaged.
|

'Managed'STA threads' that create instances of 'COM objects' that 'rely on
the finalizer' to release the COM objects instances must pump the message
queue. These are the three important pre-conditions. You can pump the queue
by calling a pumping wait like WaitOne, Join and a couple of other, you
should pump to prevent the finalizer to block, that means you should pump
regularly but not constantly.
Anyway it's better not to rely on the finalizer to release COM instances at
all, note the the MTA doesn't have this issue, so a much better solution is
to use 'free' threaded or 'both' on MTA threads.

Willy.

 

[ajbecker]

Willy,
You've been very helpful, I appreciate all the suggestions and
knowledge you've shared. Unfortunately, pumping the message queue
doesn't seem to be working. I'm thinking that the reason for that is
because my situation is calling from COM Object A to .NET code to COM
Object B. I'm wondering if when I call Join(0), it's pumping the
message queue of COM Object A not COM Object B. With my lack of
understanding of how the apartments and marshaling works, I could be
way off, but that's my best guess.

 

[Willy Denoyette [MVP]]

| Willy,
| You've been very helpful, I appreciate all the suggestions and
| knowledge you've shared. Unfortunately, pumping the message queue
| doesn't seem to be working. I'm thinking that the reason for that is
| because my situation is calling from COM Object A to .NET code to COM
| Object B. I'm wondering if when I call Join(0), it's pumping the
| message queue of COM Object A not COM Object B. With my lack of
| understanding of how the apartments and marshaling works, I could be
| way off, but that's my best guess.
|

This is a somewhat sepcial case, but I would love to see some questions
answered first.

Your COM client (native code) creates a .NET object through COM interop
which at it's turn creates a native COM object B, right?

Q1. What kind of COM client is this? what apartment does it's
creating/calling thread live in?
If it's a VB6 client, the answer is STA.
If it's not VB the apartment is ......
Q2. What kind of COM object is B, what's his apartment (ThreadingModel)
requirements? Free, Both, Apartment or Single?


Willy.

 

[ajbecker]

I will answer your questions as best I can. I'm still in the process
of trying to understand COM threading models. The COM client which
creates the .NET object has ThreadingModel = Both. The other COM
object(s) (the one(s) that the .NET object creates has ThreadingModel =
Apartment.

From what I can tell, the first client COM object was created from a

thread that was CoInitialize'd as apartment threaded. To also support
that, debuging in the .NET code shows that the thread(s) are running in
STA mode. So if I understand everything correct, we're calling:

[COM running in STA but allows both] ==> [.NET running in STA but
allows both] ==> [COM running in STA and only allows STA]

 

[Willy Denoyette [MVP]]

|I will answer your questions as best I can. I'm still in the process
| of trying to understand COM threading models. The COM client which
| creates the .NET object has ThreadingModel = Both. The other COM
| object(s) (the one(s) that the .NET object creates has ThreadingModel =
| Apartment.
|
| >From what I can tell, the first client COM object was created from a
| thread that was CoInitialize'd as apartment threaded. To also support
| that, debuging in the .NET code shows that the thread(s) are running in
| STA mode. So if I understand everything correct, we're calling:
|
| [COM running in STA but allows both] ==> [.NET running in STA but
| allows both] ==> [COM running in STA and only allows STA]
|

Ok, so we have this:

[1]COM STA thread => CCW -> [2].NET STA -> RCW => [3]COM STA

That means that all objects live in the same apartment and are entered by
the one and only Client thread (the one calling CoInitialize), that also
means that this is the thread that should spin the message loop at least
until the finalizer thread has run the "finalize" on the RCW.
However, when I re-read your other reply this isn't exactly what's happening
(please correct me if I'm wrong).
The COM client spins a thread, joins an STA, creates a .NET "COM" object,
calls methods on it, releases the instance and exits the thread proc. At
that moment the thread ceases to exist and so does the message pump, the
finalizer running (non-deterministically) in a MTA has no chance at all to
ever run the "finalize". That means, resource leaks.
The most obvious solution is to le the client thread join the MTA
(CoInitialize(..MULTI_THREADED....), the only drawback is that you will
incur marshaling overhead when transitioning from the MTA to the STA object
[3]
If you can't afford this overhead (after profiling/measuring it's impact),
you will have to implement deterministic destruction of the .NET object [2],
for instance by implementing IDisposable, and have the client to call
Dispose() before releasing the object.

Willy.

 

[ajbecker]

Here's the overall picture: The app starts and launches X STA
threads (X is variable depending on settings in the registry and the
number of CPUs on the machine.) There is also an MTA that is created
for particular objects, but does not come into play in this situation.
Each of those X STA threads will receive messages (round robin) from an
MSMQ. When one of the STAs receives a message from the MSMQ, it will
get the already cocreated .NET object, or if it hasn't been cocreated
it will cocreate it then. next, it will call one particular method on
the .NET object, and one of the parameters passed is the message that
was fetched. Then, the .NET object will look at the message and
determine what other .NET method needs to be called. It will call upon
the other method via an invoke. Within the invoked method (still
..NET,) many COM objects may be used (MOST STA, but rare situations
where freethreaded only ones are used), therefore, several RCWs may be
generated... could be zero, could be 500. When that function is
returned from, the results are returned back to the calling COM object,
a little more work is done, and then the thread goes back to listening
to the MSMQ for the next request.

Normally, there is no memory bloat. The problem comes when the process
is hit with a large number of requests on the MSMQ very very rappidly
(which is what it's designed to do.) If the X STA threads that are
spun up are limited to just 1 thread (via a change in the registry,)
then the problem doesn't happen. Everything gets released no matter
how many messages get thrown into the mix. If the number of threads is
more than 1, then we start running into the problem. Once the memory
bloat has started, every single request (that uses COM objects in the
..NET invoke) will causes additional bloat, which also supports that
the finalizer thread is blocked.

To verify your statement that everything is running on the same thread,
I went into debug and noted that they all, in fact, are running the
same thread. Also, to clear up potential confusion, in my tests, none
of the COM objects being created in the .NET invoke are going into the
MTA.

Now, just as I thought I was starting to understand the problem, I ran
this test yesterday afternoon: I ran a CurrentThread.Join(100) in
hopes that it would completely clear up any window messages that wanted
to free the COM objects. The bloat still occurs. So, when the thread
returns to the calling COM object, I had it assert any time that there
were pending message in the queue. it never asserts, but the bloat
still occurs. It's almost as though there is something else blocking
the finalizer thread. I don't think it's any particular COM object
that blocks when trying to release, because the bloat occurs when any
one of the tested COM objects is isolated. I've made it so that none
of the .NET objects I'm using are calling their own finalizer, so
unless Microsoft wrote it, there's nothing there.

As for your suggestion to let the client thread join the MTA, there are
a lot of dependencies upon being isolated in the STA. It would be like
saying, I want to own beach front property so i'll move the moon closer
to earth; there would be enormous overhead to changing the program to
do that, and there would be lots and lots of repercussions.

If you have any other suggestions, I would greatly appreciate them,
however, you've spent quite some time helping, and I would entirely
understand if you couldn't.

 

[Willy Denoyette [MVP]]

| Here's the overall picture: The app starts and launches X STA
| threads (X is variable depending on settings in the registry and the
| number of CPUs on the machine.) There is also an MTA that is created
| for particular objects, but does not come into play in this situation.
| Each of those X STA threads will receive messages (round robin) from an
| MSMQ. When one of the STAs receives a message from the MSMQ, it will
| get the already cocreated .NET object, or if it hasn't been cocreated
| it will cocreate it then. next, it will call one particular method on
| the .NET object, and one of the parameters passed is the message that
| was fetched. Then, the .NET object will look at the message and
| determine what other .NET method needs to be called. It will call upon
| the other method via an invoke. Within the invoked method (still
| .NET,) many COM objects may be used (MOST STA, but rare situations
| where freethreaded only ones are used), therefore, several RCWs may be
| generated... could be zero, could be 500. When that function is
| returned from, the results are returned back to the calling COM object,
| a little more work is done, and then the thread goes back to listening
| to the MSMQ for the next request.
|
| Normally, there is no memory bloat. The problem comes when the process
| is hit with a large number of requests on the MSMQ very very rappidly
| (which is what it's designed to do.) If the X STA threads that are
| spun up are limited to just 1 thread (via a change in the registry,)
| then the problem doesn't happen. Everything gets released no matter
| how many messages get thrown into the mix. If the number of threads is
| more than 1, then we start running into the problem. Once the memory
| bloat has started, every single request (that uses COM objects in the
| .NET invoke) will causes additional bloat, which also supports that
| the finalizer thread is blocked.
|
| To verify your statement that everything is running on the same thread,
| I went into debug and noted that they all, in fact, are running the
| same thread. Also, to clear up potential confusion, in my tests, none
| of the COM objects being created in the .NET invoke are going into the
| MTA.
|
| Now, just as I thought I was starting to understand the problem, I ran
| this test yesterday afternoon: I ran a CurrentThread.Join(100) in
| hopes that it would completely clear up any window messages that wanted
| to free the COM objects.

It makes no sense to call Join() in your managed code, it's up to the
(unmanaged code) thread that entered the STA (the unmanaged thread) to spin
a message loop. The finalizer thread marshals the call and runs the finalize
methods on this thread, if however this thread doesn't spin a message loop,
or when it's blocked (in unmanaged or managed code)for another reason like a
call to Sleep() or WaitForSingleObject and friends, the finalizer will block
as well.
So it's up to you to inspect your code to find out whether this is not the
case.

Willy.