PLS HELP - Forcing Gobal Garbage Collection

  • Thread starter Thread starter MuZZy
  • Start date Start date
M

MuZZy

Hi,

I got a situation here :) an dwonder if someone can help me.
I have an MDI app and say, the child form has a button, clicking on which calls a database
transaction - grabbing a lot of tables to the form's TDataSet variable. One transaction grabs about
10 or more Mb of data.
If i do several transactions, the memory occupied by the app goes say goes up to 180Mb - i watch it
in Task Manager.

Now i close the child form, and ChildForm.Dispose calls DataSet.Dispose(). Nothing happens, memory
allocated is still 180 Mb.
On the Parent MDI form i cick a menu option calling GC.Collect(). Again, nothing changes... still 180Mb

Now i leave the program running 'idle' (app does nothing) and i start doing some charts in Excel and
Word for some other project. Suddenly in about 5 min i look at Task Manager and bummmm - my running
application now occupies only 5(five!!!) MB of memory... instead of 180 Mb

I realize that garbage collection doesn't do the job right away, but only when it considers it
nessesary, but is there a way to force it somehow? We have clients complaining that program is
crashing on them because of insufficient memory when they do many transactions.

Any suggestions would be highly appreciated!!!

Thank you,
Andrey
 
MuZZy said:
Hi, Hi.
I got a situation here :) an dwonder if someone can help me.
I have an MDI app and say, the child form has a button, clicking on which calls a database transaction - grabbing a lot of tables
to the form's TDataSet variable. One transaction grabs about 10 or more Mb of data.
If i do several transactions, the memory occupied by the app goes say goes up to 180Mb - i watch it in Task Manager.

Now i close the child form, and ChildForm.Dispose calls DataSet.Dispose(). Nothing happens, memory allocated is still 180 Mb.
On the Parent MDI form i cick a menu option calling GC.Collect(). Again, nothing changes... still 180Mb

Now i leave the program running 'idle' (app does nothing) and i start doing some charts in Excel and Word for some other project.
Suddenly in about 5 min i look at Task Manager and bummmm - my running application now occupies only 5(five!!!) MB of memory...
instead of 180 Mb

I realize that garbage collection doesn't do the job right away, but only when it considers it nessesary, but is there a way to
force it somehow? We have clients complaining that program is crashing on them because of insufficient memory when they do many
transactions.

Any suggestions would be highly appreciated!!!
Click to expand...

One approach would be to rework your data requests to not
get so much. This would require some more work on your
part, but would improve the performance of your app. Have
you ever tried to deal with 10 MB of data in a form? Surely
nobody needs to scroll thru it all.
 
MuZZy,

When you have clients complaining about that, looking at the Task
Manager is the last thing that you want to do. It is possible that you are
simply loading too much information into the local client process and that
you should look into changing your implementation.

Basically, what that error means is that you have references to objects
that are taking up that much memory, and keeping them around somehow. As
you have shown, the CLR will give up memory when the system asks for it.

How large are the datasets that you are loading? Is there a need to
keep that around all at one time? Can you not break it down into smaller
units of work? Doing so will probably require more trips to the DB server,
but in the end, I can't imagine that your system is running to efficiently
due to the immense amounts of memory it is taking up.
 
Larry said:
One approach would be to rework your data requests to not
get so much. This would require some more work on your
part, but would improve the performance of your app. Have
you ever tried to deal with 10 MB of data in a form? Surely
nobody needs to scroll thru it all.
Click to expand...

Well, it's not 10 Mb of data shown on the form. It's stored in a TDataSet variable.
The approach to rework the ddata requests might work if the app was in the early development stage.
Now it's the whole framework and there is no way to change the architecture - it might require about
a year of work by 20 programmers :)

My question is more about how to fix what i already have, how to force GC to do the collection.

Andrey
 
You can force GC collection by calling GC.Collect(). But that most
probably won't solve your memory problems. If GC.Collect() has no
effect, that means that you are still maintaining references to objects
and that's why it is not able to collect.

Regards
Senthil
 
sadhu said:
You can force GC collection by calling GC.Collect(). But that most
probably won't solve your memory problems. If GC.Collect() has no
effect, that means that you are still maintaining references to objects
and that's why it is not able to collect.

Regards
Senthil
Click to expand...

That's the problem! The object consuming the memory belongs to the MDI child form, and when the from
is closed, it suppose to break references to the object as well, right? But memory is stil not
released...
 
Maybe there is a reference to that object somewhere else? Is there a
property/method that returns it? Try setting the object to null before
calling GC.Collect(). Try using the CLR profiler to get a dump of the
existing objects.

Regards
Senthil
 
MuZZy said:
Larry Brasfield wrote: ....

Well, it's not 10 Mb of data shown on the form. It's stored in a TDataSet variable.
The approach to rework the ddata requests might work if the app was in the early development stage.
Now it's the whole framework and there is no way to change the architecture - it might require about a year of work by 20
programmers :)

My question is more about how to fix what i already have, how to force GC to do the collection.
Click to expand...

Of course, I am in no position to make that sort of
judgment. I would suggest, however, that you be
sure to consider that the number of rows that will
be retrieved by your existing queries is likely to
grow over time. If that growth is slower than the
capacity of computers your clients are able and
willing to procure, maybe that growth is OK.
 
Nicholas said:
MuZZy,

When you have clients complaining about that, looking at the Task
Manager is the last thing that you want to do. It is possible that you are
simply loading too much information into the local client process and that
you should look into changing your implementation.

Basically, what that error means is that you have references to objects
that are taking up that much memory, and keeping them around somehow. As
you have shown, the CLR will give up memory when the system asks for it.

How large are the datasets that you are loading? Is there a need to
keep that around all at one time? Can you not break it down into smaller
units of work? Doing so will probably require more trips to the DB server,
but in the end, I can't imagine that your system is running to efficiently
due to the immense amounts of memory it is taking up.
Click to expand...

Ok, i guess i made a bad explanation of the problem.. Will try again - Say, i have a form (Form1)
with one button, pressing which creates another form(Form2), which actually makes db transactions
and stores results in it's DataSet object. When the from Form2 is closed, it suppose to release
memory held by the DataSet variable, which in fact doesn't happen. If you Click on button on form1
-> open Form2->do a transaction -> Close Form2, and repeat this several times, you will see that
the memory allocated ini each transaction just summed, and you have closed Form2 but all the
transactions are still somewhere in the memory. GC.Collect() doesn't do any good...

Then, at some point, it may drop (garbage collection starts working), but sometimes too late.
 
The GC only collects the managed heap memory portion, so before you say the
GC doesn't it job you should take a look at the size of the managed heap
using perfmon (check the CLR memory counters). Task manager only shows the
working set and virtual memory counters, I suggest you run perfmon when
peeking into memory used by your processes (managed/unmanaged).
An application (any "windows" application) will see it's working set reduced
when idle (same when minimized), that means all but the minimum WS data
pages are written to the page file. This is an OS feature that exists since
years on all windows versions.

Willy.
 
Willy said:
The GC only collects the managed heap memory portion, so before you say the
GC doesn't it job you should take a look at the size of the managed heap
using perfmon (check the CLR memory counters). Task manager only shows the
working set and virtual memory counters, I suggest you run perfmon when
peeking into memory used by your processes (managed/unmanaged).
An application (any "windows" application) will see it's working set reduced
when idle (same when minimized), that means all but the minimum WS data
pages are written to the page file. This is an OS feature that exists since
years on all windows versions.
Click to expand...

Well, if my app is managed, doesn't it mean that the objets used are in the managed heap?
Ok, i made a further investigation using CLR Profiler. When i closed all froms which were making
transaktions,other words consuming memory, th eprofiler showed me that there are still Form2 class
objects and DataSet objects in the memory. And only in 10 min of idle app running, those objects
started disappearing - i was re-launching heap map of profiler and saw some objects disappearing...
 
I hope you are explicitly closing the Form using Close(), otherwise
resources associated with that form won't be disposed (until it gets
finalized).
 
MuZZy said:
Ok, i guess i made a bad explanation of the problem.. Will try again -
Say, i have a form (Form1) with one button, pressing which creates another
form(Form2), which actually makes db transactions and stores results in
it's DataSet object. When the from Form2 is closed, it suppose to release
memory held by the DataSet variable, which in fact doesn't happen. If you
Click on button on form1 -> open Form2->do a transaction -> Close Form2,
and repeat this several times, you will see that the memory allocated ini
each transaction just summed, and you have closed Form2 but all the
transactions are still somewhere in the memory. GC.Collect() doesn't do
any good...

Then, at some point, it may drop (garbage collection starts working), but
sometimes too late.
Click to expand...

As long as you are measuring memory consumption using the right tools, you
are simply guessing.
First you have to make sure you excessive memory consumption is managed
memory.
If it is, you need to run the application through a managed memory profiler
in order to investigate your object allocation scheme and find the objects
references that are responsible for the objects not being eligible for GC.
If huge memory consumption is not due to managed heap growth, you should
investigate your unmanaged resource acquisition and release pattern, that
is, check if your Dispose your disposable objects.

Willy.
 
Willy said:
As long as you are measuring memory consumption using the right tools, you
are simply guessing.
First you have to make sure you excessive memory consumption is managed
memory.
If it is, you need to run the application through a managed memory profiler
in order to investigate your object allocation scheme and find the objects
references that are responsible for the objects not being eligible for GC.
If huge memory consumption is not due to managed heap growth, you should
investigate your unmanaged resource acquisition and release pattern, that
is, check if your Dispose your disposable objects.

Willy.
Click to expand...

I don't use any unmanaged resources and i know for sure that memory waste is caused by DataSet and
ArrayList objects belonging to a form not being garbage collected in time after i close the form.

I ran the app thru the Profiler and it showed me those objects (with sizes of 14 Mb, 20Mb, etc) in
the map after i disposed the from.
If i wait about 10 min and check the map again, most of the objects are gone. That's how i assume
that GC doen't collect them promptly/timely...

Thank you,
Andrey
 
I don't use any unmanaged resources and i know for sure that memory waste
is caused by DataSet and ArrayList objects belonging to a form not being
garbage collected in time after i close the form.

I ran the app thru the Profiler and it showed me those objects (with sizes
of 14 Mb, 20Mb, etc) in the map after i disposed the from.
If i wait about 10 min and check the map again, most of the objects are
gone. That's how i assume that GC doen't collect them promptly/timely...

Thank you,
Andrey
Click to expand...

So you mean you have objects with sizes of > 10MB, that means they live in
the Large Object Heap (LH).
The LH heap is something special in that:
1. It get's swept only with a full GC run.
2. It doesn't get compacted, that means it can suffer from heap
fragmentation.
3. The unused memory pages are ONLY returned to the OS when there is memory
pressure AND the CLR is willing to reduce it's LH segment size, or when the
OS memory manager decides to trim the WS (when the application is idle or
ample memory available), NOT by a GC run.

So again I suggest you take a look at the CLR memory perf. counters and
watch the Gen0, Gen1, Gen2 and LH size counters and watch you allocation
pattern.

Another question, you say you don't use any unmanaged resources, but what is
the data source for your DataSet? Is it a DB connection or any file data? If
the answer is yes, well there you have your unmanaged resources. The
question is do they get released by calling Dispose (same for your DataSet)
or are you simply waiting for the finalizer to run?



Willy.
 
Also keep in mind that the default set up of the task manager is to show
ONLY the physical memory used by your application, to go along with what
Willy said below. If you want to see how much is being used (and still task
manager isn't the best place for this) I would suggest adding Virtual Memory
Size to the task manager.

I knew about the minimize, but never realized windows swapped it out if it
was idle for some time, HMMM, going to have to play with that tonight.

--
Thanks
Wayne Sepega
Jacksonville, Fl


"When a man sits with a pretty girl for an hour, it seems like a minute. But
let him sit on a hot stove for a minute and it's longer than any hour.
That's relativity." - Albert Einstein
 
Wayne said:
Also keep in mind that the default set up of the task manager is to show
ONLY the physical memory used by your application, to go along with what
Willy said below. If you want to see how much is being used (and still task
manager isn't the best place for this) I would suggest adding Virtual Memory
Size to the task manager.

I knew about the minimize, but never realized windows swapped it out if it
was idle for some time, HMMM, going to have to play with that tonight.
Click to expand...

Actually for it to happen not only the app needsto be idle, but you need to switch to another app
and do something...
 
Wayne said:
Also keep in mind that the default set up of the task manager is to show
ONLY the physical memory used by your application, to go along with what
Willy said below. If you want to see how much is being used (and still
task
manager isn't the best place for this) I would suggest adding Virtual
Memory
Size to the task manager.

I knew about the minimize, but never realized windows swapped it out if it
was idle for some time, HMMM, going to have to play with that tonight.
Click to expand...

The WS manager only trims when ample memory is available, the WS are not
touched if plenty of memory is free, this would definitely be a waste of
time and resources. Minimizing a windows app. is a special case, here it's
supposed you won't be using the application for a long time.
Note also that the WS manager selects the least recently used pages to be
removed first, this removal is not per process but global. So it's possible
that you don't see a change in WS size of you windows application, while
other processes who have their pages loaded during the boot process have
their WS size reduced to 0 (f.I Winlogon.exe)

Willy.
 
Whoa! I'm having the exact same problem and it does involve datasets as well.
I know that the Dispose is called on my OleConnection due to my use of the
using() as follows:

public static DataSet ExecuteDataset(...)
{
//create & open a OleDbConnection, and dispose of it after we are done.
using (OleDbConnection cn = new OleDbConnection())
{
cn.ConnectionString = connectionString;
cn.Open();

//call the overload that takes a connection in place of the connection
string
return ExecuteDataset(cn, commandType, commandText, commandParameters);
}
}

But I am to call the Dispose() on the returned DataSet right when I believe
I'm done with it?
 
Back
Top