Pressuring GC to give up clean up memory.

[Frank Rizzo]

I have an app ( referenced in thread
news://msnews.microsoft.com:119/[email protected] )
that won't give up memory. Several people stated that GC won't clean up
memory unless pressured to do so (e.g. system running out of memory).
So I wrote a quickie app to gobble up memory 100mb at a time, but I
still don't see anything giving up memory (including other .NET apps).
I see the commit charge going up and go past the physical memory, but
still no dice.

Did I write it correctly?

using System;
using System.Collections.Generic;
using System.Text;
using System.Collections;

namespace ConsoleApplication1
{
class Program
{
static void Main(string[] args)
{
ArrayList lst = new ArrayList();
int count = 0;

try
{
while (true)
{
count++;

// allocate 1 MB
Byte[] K = new byte[1024 * 1024];
lst.Add(K);

Console.WriteLine(count);

if (count % 100 == 0)
Console.ReadKey() ;
}
}
catch (Exception ex)
{
Console.WriteLine(GC.GetTotalMemory(false).ToString());
Console.WriteLine(ex.ToString());
Console.ReadLine();
throw;
}
}
}
}

 

[Nicholas Paldino [.NET/C# MVP]]

Frank,

Is the number you are seeing the only concern, or is the memory that is
being consumed by your app having a measurable effect on other applications
on the box?

 

[Frank Rizzo]

Frank,

Is the number you are seeing the only concern, or is the memory that is
being consumed by your app having a measurable effect on other applications
on the box?

Basically, the powers that be, decided that the app is going to be on
the same box as the SqlServer2k5 64-bit, which consumes ungodly amounts
of memory (compared to sqlserver2k, anyways). So the sum of my app and
SQL Server is pushing up against the limits of physical memory. When it
does cross that limit, things get really, really slow. When my app is
idle for more than a certain amount of time, I try to dump the memory
contents by setting the root object to null - but that does not seem to
be having an effect.

 

[Nicholas Paldino [.NET/C# MVP]]

Frank,

Far be it for me to challenge the powers that be, but is your app
dependent on running on the same exact box as sql server? As a general
guideline, I don't run ANYTHING else on SQL Server boxes that doesn't need
to be there.

 

[Frank Rizzo]

Frank,

Far be it for me to challenge the powers that be, but is your app
dependent on running on the same exact box as sql server? As a general
guideline, I don't run ANYTHING else on SQL Server boxes that doesn't need
to be there.

I couldn't agree with you more. No, the app does not depend on having
sql server on the same box. I think it is the issue of cost (e.g.
requiring customers to buy another box, another set of win2k3 licenses,
etc...).

Anyway, getting back to the original problem: would the code in the top
node of the thread increase pressure on GC?

Regards

 

[Chris Mullins [MVP]]

Basically, the powers that be, decided that the app is going to be on the
same box as the SqlServer2k5 64-bit, which consumes ungodly amounts of
memory (compared to sqlserver2k, anyways).

Change the configuration of SQL 2005 from "Use All Memory" (which is the
default) to something else.

 

[Nicholas Paldino [.NET/C# MVP]]

Frank,

Well, yes, it would, but at the cost of increasing the memory of the app
that you are running. It will be a zero-sum game at that point, with no
gains or loss overall.

What you want to do is call the SetProcessWorkingSetSizeEx (along with
the process handle) in order to reduce the working set of your app.

 

[Alun Harford]

I have an app ( referenced in thread
news://msnews.microsoft.com:119/[email protected] )
that won't give up memory. Several people stated that GC won't clean up
memory unless pressured to do so (e.g. system running out of memory). So
I wrote a quickie app to gobble up memory 100mb at a time, but I still
don't see anything giving up memory (including other .NET apps).
I see the commit charge going up and go past the physical memory, but
still no dice.

Did I write it correctly?

Well the garbage collector won't clear up memory if there's a reference
to it.

while (true)
{
count++;

// allocate 1 MB
Byte[] K = new byte[1024 * 1024];
lst.Add(K);

You're adding the byte[] to the list, so there's still a reference around.
If you leave out the last line, GC will give up memory when the system
needs it.

Alun Harford

 

[J.Marsch]

Frank:

Another thing to keep in mind is that the garbage collector is only
triggered when your app makes an allocation (well, when you allocate memory
and a GC is required, a GC occurs).

So, if your app is sitting idle while you are applying memory pressure from
another application, you probably won't see a GC occur until/unless your app
tries to allocate memory.

Here is a link to the post where I learned about the GC-on-allocate
behavior:
http://blogs.msdn.com/tess/archive/2007/04/10/net-garbage-collector-popquiz-followup.aspx

(that's a really good blog, incidentally)

 

[Frank Rizzo]

Change the configuration of SQL 2005 from "Use All Memory" (which is the
default) to something else.

Unfortunately that can only happen over DBA's dead body. That's a
direct quote from him.

 

[Frank Rizzo]

I have an app ( referenced in thread
news://msnews.microsoft.com:119/[email protected]
) that won't give up memory. Several people stated that GC won't
clean up memory unless pressured to do so (e.g. system running out of
memory). So I wrote a quickie app to gobble up memory 100mb at a time,
but I still don't see anything giving up memory (including other .NET
apps).
I see the commit charge going up and go past the physical memory, but
still no dice.

Did I write it correctly?

Well the garbage collector won't clear up memory if there's a reference
to it.

while (true)
{
count++;

// allocate 1 MB
Byte[] K = new byte[1024 * 1024];
lst.Add(K);

You're adding the byte[] to the list, so there's still a reference around.
If you leave out the last line, GC will give up memory when the system
needs it.

No, no, no. I think everyone misunderstood the point of this code.
This code is a standalone console app that is meant to gobble up memory
in order to induce other .NET applications to start its garbage collection.

 

[Steven Cheng[MSFT]]

Hi Frank,

For the .NET managed application, the memory pressure(which GC care) is
about the managed clr heap memory(which consume part of the OS process's
virtual memory space). You do not need to use other process(application) to
add memory pressure to a separate process(application).

Also, if you want to trace .NET memory behavior, you should use performance
counters and monitor the .net CLR managed heap size. You can use
perfmon.exe and choose the ".NET CLR memory" performance object and choose
some counters such as "byte in all heaps" and watch it when your
application is running. Normally, this size will be adjusted when there is
managed objects be collected(GC).

BTW, I'll also update you in your former thread about SMO objects.

Sincerely,

Steven Cheng

Microsoft MSDN Online Support Lead


This posting is provided "AS IS" with no warranties, and confers no rights.

 

[Laura T.]

Have you tried GC.AddMemoryPressure()?
It usually gives some kicks to GC.
GC has a little nice algorithm to check if it needs really to GC something.
One of them, it seems, is global system paging activity, and also your
processes generated paging.
Even if you allocate a lot, but the system has no activity, GC doesn't see
any reason to release something (you might need it 2 seconds later, GC
hopes). If the system activity level goes up, and paging goes wild, GC
starts working.. really.

 

[Willy Denoyette [MVP]]

Basically, the powers that be, decided that the app is going to be on the
same box as the SqlServer2k5 64-bit, which consumes ungodly amounts of
memory (compared to sqlserver2k, anyways). So the sum of my app and SQL
Server is pushing up against the limits of physical memory. When it does
cross that limit, things get really, really slow. When my app is idle for
more than a certain amount of time, I try to dump the memory contents by
setting the root object to null - but that does not seem to be having an
effect.

If SQL Server is configured to consume exactly what it needs, and there is
enough memory left to let your application run without the risk of excessive
paging you are set, else you are in a world of pain, playing ticks with the
GC won't help. Just face it, when there is memory pressure (that is, you
have reached a certain threshold) because SQL needs more memory, the OS will
ask your application (and others) to reduce it's working set, the CLR will
trigger a full collection and the CLR's memory manager will return freed
heap space (if any) to the OS, if the GC cannot release enough memory, the
OS will simply trim your application's (and others) working set, by writing
modified pages to the page file, until there is enough free memory to
fulfill SQL's request. Whenever your application get's a chance to run
again, it will "possibly" need to page-in the pages previously written to
the page file, but to make this happen, the OS will need to trim the WS of
SQL server, this is especially bad from a performance perspective, so your
only options are:
- Reduce the memory consumption of your application, inspect your object
hierarchies, are you using the most optimal container types, are you taking
care of the growth algorithms of some of these containers, aren't you
overusing OO paradigms?
- If your application runs as 64 bit application, recompile it as 32 bit
(unless you need to address more than 2GB), 64 BIT applications are real
memory hogs, especially managed applications.
- Add more memory to the box.

Willy.

 

[Alvin Bruney [MVP]]

Seems to me that this is a good case for requesting another machine. Either
way is painful but requesting a new machine to run the app is easier because
it is a one-time cost where as tweaking the application is not.

--
Regards,
Alvin Bruney
------------------------------------------------------
Shameless author plug
Excel Services for .NET is coming...
https://www.microsoft.com/MSPress/books/10933.aspx
OWC Black Book www.lulu.com/owc
Professional VSTO 2005 - Wrox/Wiley

 

[Willy Denoyette [MVP]]

Seems to me that this is a good case for requesting another machine.
Either way is painful but requesting a new machine to run the app is
easier because it is a one-time cost where as tweaking the application is
not.

Sure, in the case of a DB server where the DBA (rightfully?) fails to
partition the workload and assumes every single byte of memory to be
available for the DB, there is nothing else you can do, other than ask the
customer to add another box. Things got worse when running 64 bit versions
of SQL, which consumes a lot more memory than the 32 bit version, running
another application on the same box that also consumes a lot of memory is
simply asking for troubles, the DBA will point to the application when
things turn bad, the application dev. will point to the DB for the same
reason. Management doesn't understand the issue, they think they bought kind
of a 64 bit mainframe, they will point the application dev when things
turn-out wrong, see the picture...


Willy.