GC problem with large object - .net 1.1

[Rolf Welskes]

Hello,
I have an applications witch works with bitmaps.
Such bitmaps are in windows (forms) sizeable. Each time it is neccary to
allocate new bitmap with any size-step, because the bitmap
is generated.

So there is the following situation under .net 1.1

If the window (bitmap) is sized for exampel 10 bitmaps are generated and
disposed in 2 seconds.
Because the GC does not make garbage collection directly, this costs 10 MB
ram instead of 1.

If it is exreme, an application which needs realy 10 MB - you see in the
taskmanager 50 MB.

But this is the first layer of the problem.

Now some time later, maybe the GC makes a garbace collection.
Now GC-Memory has 10 MB used and 40 MB free.
BUT the GC does not give the memory back to the system.
(sometimes GC gives a little back to the system, but not all).
Means: for a long time you see 50 MB RAM resources used in the tasks manger,
what is not the reality, it's only 10 MB.

This problem is very hard, because customers see this in the task manager
und do not buy the product.

So question:
Is there any way under dot net 1.1 (.net net 2.0 is absolute impossible in
the moment because of many reasons),
to reduce the shown used Memory, which is wrong in the taskmanager for this
app.

Means is it possible to solve the problem when the bitmap is sized that
memory is not accumulated, because
with each new bitmap the old is disposed.
I think there will be no solution under .Net / C#, but would be fine.
But also a solution using C++ / unmanged code would be nice. (for the
allocation of memory of the bitmaps only).

Thank you for any help.

Rolf Welskes

 

[Barry Kelly]

I have an applications witch works with bitmaps.
Such bitmaps are in windows (forms) sizeable. Each time it is neccary to
allocate new bitmap with any size-step, because the bitmap
is generated.

You could consider using a larger bitmap than required and only using a
portion of it. That would avoid constantly resizing it for any size
step.

So there is the following situation under .net 1.1

If the window (bitmap) is sized for exampel 10 bitmaps are generated and
disposed in 2 seconds.

Because the GC does not make garbage collection directly, this costs 10 MB
ram instead of 1.

Are you really disposing the objects? That is, are you calling .Dispose
on the IDisposable Image, or using 'using' to limit the scope of the
images, etc.? When processing images etc., I *never* rely on the GC to
dispose of objects due to the working set peaks that causes.

Now some time later, maybe the GC makes a garbace collection.
Now GC-Memory has 10 MB used and 40 MB free.
BUT the GC does not give the memory back to the system.
(sometimes GC gives a little back to the system, but not all).
Means: for a long time you see 50 MB RAM resources used in the tasks manger,
what is not the reality, it's only 10 MB.

As you probably know, this is largely irrelevant. The numbers you can
find under Task Manager are either going to be "VM Size" (Private Bytes,
actual non-shared pages ultimately allocated using VirtualAlloc) or "Mem
Usage" (Working Set, the amount of memory that Windows has allocated to
the process for the moment, because that's how much it needs, under the
balance of current conditions and competing uses for the same memory, to
run without swapping). Many non-managed applications don't decrease
their private bytes unless they've implemented a custom memory manager.
Working set isn't going to decrease unless some event occurs which
causes the OS to recalculate the working set.

You've got to accept that "Free Memory" under Task Manager is actually
wasted memory. The ideal situation is for all memory to be allocated, if
not by processes, then by the system cache.

This problem is very hard, because customers see this in the task manager
und do not buy the product.

By that logic, people shouldn't run Firefox either, since it regularly
wanders up into 200 MB+ working set on my machine!

So question:
Is there any way under dot net 1.1 (.net net 2.0 is absolute impossible in
the moment because of many reasons),
to reduce the shown used Memory, which is wrong in the taskmanager for this
app.

Means is it possible to solve the problem when the bitmap is sized that
memory is not accumulated, because
with each new bitmap the old is disposed.
I think there will be no solution under .Net / C#, but would be fine.
But also a solution using C++ / unmanged code would be nice. (for the
allocation of memory of the bitmaps only).

You could investigate calling SetProcessWorkingSetSize() with a value of
-1. As the documentation says, this will cause the app to swap out to
disk.

-- Barry

 

[Jeffrey Tan[MSFT]]

Hi Rolf,

How about this issue now? I am monitoring this thread for your feedback
yet. Besides Barry's reply, I want to add some extra comment:

Can you tell me how do you tell me which value do you see in TaskManager
for 50MB? Normally there are 2 memory related values: "Mem Usage" and "VM
Size".

Just as Barry pointed out, "VM Size" is the actually "Private Bytes", which
is non-shared virtual memory allocated by your application, while "Mem
Usage" is "Working Set", which is the portion of physical memory that your
application resides in RAM(consider there are much memory that is swapped
out to disk pagefile). These 2 somewhat misleading names are changed to
"Private Bytes" and "Working Set" in Vista.

From programmer's perspective, the "Mem Usage" value normally does not make
sense, since Windows Virtual Memory Manager takes care of the working set
management of each process for best performance. "VM Size" is an important
value, which may used to monitor the memory usage of your application, for
example, if this value keeps growing, it means there are certain memory
leak in your application.

The reason we should take care of "VM Size" instead of "Mem Usage" is that:
in Windows programming, user mode application can only allocate/free
virtual memory of our process, physical memory status is almost out of our
control. For example, if you minimize your GUI application to taskbar, you
will find the "Mem Usage" value to be decreased a lot while leaving the "VM
Size" without any change. As you can see, this physical memory usage really
does not make sense to programmer.

So I need your confirmation over which value you are referring to.

Second, can you tell me how do you ensure that your application is actually
using 10MB virtual/physical memory instead of 50MB?

Finally, SetProcessWorkingSetSize with both -1 values are useful to reduce
the working set to a minimum if this is what you want. Actually in .Net,
this API is encapsulated by the following 2 prperties:
System.Diagnostics.Process.MaxWorkingSet
System.Diagnostics.Process.MinWorkingSet

I will wait for your further feedback. Thanks.

Best regards,
Jeffrey Tan
Microsoft Online Community Support
==================================================
Get notification to my posts through email? Please refer to
http://msdn.microsoft.com/subscriptions/managednewsgroups/default.aspx#notif
ications.

Note: The MSDN Managed Newsgroup support offering is for non-urgent issues
where an initial response from the community or a Microsoft Support
Engineer within 1 business day is acceptable. Please note that each follow
up response may take approximately 2 business days as the support
professional working with you may need further investigation to reach the
most efficient resolution. The offering is not appropriate for situations
that require urgent, real-time or phone-based interactions or complex
project analysis and dump analysis issues. Issues of this nature are best
handled working with a dedicated Microsoft Support Engineer by contacting
Microsoft Customer Support Services (CSS) at
http://msdn.microsoft.com/subscriptions/support/default.aspx.
==================================================
This posting is provided "AS IS" with no warranties, and confers no rights.

 

[Rolf Welskes]

Hello,
thank you for your answer.
first I do now mean any memory which is displayed in the taskmanger which
show differences between virtual or not virtual memory.
I mean the number you see, when opening the taskmager, click the tab
processes, now you see the name of the process and the used memory.
This is a number which many customers check.
I think this is all used memory of the appliccation, mayby virtual or in
ram.

Furthermore:
If I see after sizing the bitmap for example 50 mb and wait some time and
then I start a simple helper-program with which I allocate (in c++) windows
memory in steps of 1 MB until the limit (the limit is when all real memory
and virtual memory on disk is at the end), I get the message out of memory
of this helper program. If I now look in task manger the application with
the bitmap has only 10 MB not 50.
So I see that the GC of the application does not give all free memory to the
windows system.

Ok, I am developer for more than 20 years, I know the windows system and the
dotnet system and I know that it is not realy a problem that I see 50 MB of
memory usage in the task manger, because if memory realy goes down it is
freed and available.
But for simple customers they think 50 MB in standby of a program for such a
little program, Catastrohic !

Because products must be sold and I think it will cost 5 or 10 years until
the normal customer knows that this memory usage numbers in task manager are
no problem, I have to find a way to make it lower.

Best would be a solution where I could declare the bitmap in a c++ code, get
the handle allocate the memory of the non-manged heap and
with this handle I would get a dotnet bitmap, use it and then call delete
bitMapMemory in c++ and memory is free.

So question is there a solution to do something like this.

Thank you and best regards
Rolf Welskes

Hi Rolf,

How about this issue now? I am monitoring this thread for your feedback
yet. Besides Barry's reply, I want to add some extra comment:

Can you tell me how do you tell me which value do you see in TaskManager
for 50MB? Normally there are 2 memory related values: "Mem Usage" and "VM
Size".

Just as Barry pointed out, "VM Size" is the actually "Private Bytes",
which
is non-shared virtual memory allocated by your application, while "Mem
Usage" is "Working Set", which is the portion of physical memory that your
application resides in RAM(consider there are much memory that is swapped
out to disk pagefile). These 2 somewhat misleading names are changed to
"Private Bytes" and "Working Set" in Vista.

From programmer's perspective, the "Mem Usage" value normally does not
make
sense, since Windows Virtual Memory Manager takes care of the working set
management of each process for best performance. "VM Size" is an important
value, which may used to monitor the memory usage of your application, for
example, if this value keeps growing, it means there are certain memory
leak in your application.

The reason we should take care of "VM Size" instead of "Mem Usage" is
that:
in Windows programming, user mode application can only allocate/free
virtual memory of our process, physical memory status is almost out of our
control. For example, if you minimize your GUI application to taskbar, you
will find the "Mem Usage" value to be decreased a lot while leaving the
"VM
Size" without any change. As you can see, this physical memory usage
really
does not make sense to programmer.

So I need your confirmation over which value you are referring to.

Second, can you tell me how do you ensure that your application is
actually
using 10MB virtual/physical memory instead of 50MB?

Finally, SetProcessWorkingSetSize with both -1 values are useful to reduce
the working set to a minimum if this is what you want. Actually in .Net,
this API is encapsulated by the following 2 prperties:
System.Diagnostics.Process.MaxWorkingSet
System.Diagnostics.Process.MinWorkingSet

I will wait for your further feedback. Thanks.

Best regards,
Jeffrey Tan
Microsoft Online Community Support
==================================================
Get notification to my posts through email? Please refer to
http://msdn.microsoft.com/subscriptions/managednewsgroups/default.aspx#notif
ications.

Note: The MSDN Managed Newsgroup support offering is for non-urgent issues
where an initial response from the community or a Microsoft Support
Engineer within 1 business day is acceptable. Please note that each follow
up response may take approximately 2 business days as the support
professional working with you may need further investigation to reach the
most efficient resolution. The offering is not appropriate for situations
that require urgent, real-time or phone-based interactions or complex
project analysis and dump analysis issues. Issues of this nature are best
handled working with a dedicated Microsoft Support Engineer by contacting
Microsoft Customer Support Services (CSS) at
http://msdn.microsoft.com/subscriptions/support/default.aspx.
==================================================
This posting is provided "AS IS" with no warranties, and confers no
rights.

 

[Jeffrey Tan[MSFT]]

Hi Rolf,

Thanks for your feedback!

By default, there will only be "Mem Usage" column in TaskManager, so I am
assuming you are concerning about this column value, that is "Working Set"
of a process.

Reducing the working set is not a trivial task, there a number of aspects
can be improved to reduce an application's working set. Also, reducing
"Working Set" can really improve the performance of overall system, this is
because as working set size increases, the application's physical memory
demand increases, which makes other applications have less RAM usage.

The common improvement to reduce working set are listed in the "Working Set
Considerations" in the article below, you may give these recommendations a
review and make some improvement in your application design and coding:
"Chapter 5 -Improving Managed Code Performance"
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnpag/html/
scalenetchapt05.asp

If you can not change the design and coding, you may at least try to use
NGen to reduce the working set. You can also use the Vadump.exe tool to
measure your application's working set size. For more information, see
"Vadump.exe: Virtual Address Dump" at:
http://www.microsoft.com/windows2000/techinfo/reskit/tools/existing/vadump-o
asp

Finally, after completing all the design/coding side improvement above, you
may try to use SWS tool written by John Robbins to smooth your application
working set at linking order and functions layout order:
"Improving Runtime Performance with the Smooth Working Set Tool "
http://msdn.microsoft.com/msdnmag/issues/1000/bugslayer/
"Improving Runtime Performance with the Smooth Working Set Tool 2"
http://msdn.microsoft.com/msdnmag/issues/1200/bugslayer/

Hope this helps.

Best regards,
Jeffrey Tan
Microsoft Online Community Support
==================================================
Get notification to my posts through email? Please refer to
http://msdn.microsoft.com/subscriptions/managednewsgroups/default.aspx#notif
ications.

Note: The MSDN Managed Newsgroup support offering is for non-urgent issues
where an initial response from the community or a Microsoft Support
Engineer within 1 business day is acceptable. Please note that each follow
up response may take approximately 2 business days as the support
professional working with you may need further investigation to reach the
most efficient resolution. The offering is not appropriate for situations
that require urgent, real-time or phone-based interactions or complex
project analysis and dump analysis issues. Issues of this nature are best
handled working with a dedicated Microsoft Support Engineer by contacting
Microsoft Customer Support Services (CSS) at
http://msdn.microsoft.com/subscriptions/support/default.aspx.
==================================================
This posting is provided "AS IS" with no warranties, and confers no rights.

 

[Rolf Welskes]

Hello,
thank you, I will look in the articles.

Thank you again.
Rolf Welskes

Hi Rolf,

Thanks for your feedback!

By default, there will only be "Mem Usage" column in TaskManager, so I am
assuming you are concerning about this column value, that is "Working Set"
of a process.

Reducing the working set is not a trivial task, there a number of aspects
can be improved to reduce an application's working set. Also, reducing
"Working Set" can really improve the performance of overall system, this
is
because as working set size increases, the application's physical memory
demand increases, which makes other applications have less RAM usage.

The common improvement to reduce working set are listed in the "Working
Set
Considerations" in the article below, you may give these recommendations a
review and make some improvement in your application design and coding:
"Chapter 5 -Improving Managed Code Performance"
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnpag/html/
scalenetchapt05.asp

If you can not change the design and coding, you may at least try to use
NGen to reduce the working set. You can also use the Vadump.exe tool to
measure your application's working set size. For more information, see
"Vadump.exe: Virtual Address Dump" at:
http://www.microsoft.com/windows2000/techinfo/reskit/tools/existing/vadump-o
asp

Finally, after completing all the design/coding side improvement above,
you
may try to use SWS tool written by John Robbins to smooth your application
working set at linking order and functions layout order:
"Improving Runtime Performance with the Smooth Working Set Tool "
http://msdn.microsoft.com/msdnmag/issues/1000/bugslayer/
"Improving Runtime Performance with the Smooth Working Set Tool 2"
http://msdn.microsoft.com/msdnmag/issues/1200/bugslayer/

Hope this helps.

Best regards,
Jeffrey Tan
Microsoft Online Community Support
==================================================
Get notification to my posts through email? Please refer to
http://msdn.microsoft.com/subscriptions/managednewsgroups/default.aspx#notif
ications.

Note: The MSDN Managed Newsgroup support offering is for non-urgent issues
where an initial response from the community or a Microsoft Support
Engineer within 1 business day is acceptable. Please note that each follow
up response may take approximately 2 business days as the support
professional working with you may need further investigation to reach the
most efficient resolution. The offering is not appropriate for situations
that require urgent, real-time or phone-based interactions or complex
project analysis and dump analysis issues. Issues of this nature are best
handled working with a dedicated Microsoft Support Engineer by contacting
Microsoft Customer Support Services (CSS) at
http://msdn.microsoft.com/subscriptions/support/default.aspx.
==================================================
This posting is provided "AS IS" with no warranties, and confers no
rights.

 

[Yan-Hong Huang[MSFT]]

Hi Rolf,

Thanks for your update. Jeffery is out of office today. If there is any
questions after you read the article, please feel free to post a new thread
or reply here. We are glad to work with you.

Have a good day.

Sincerely,
Yanhong Huang
Microsoft Online Community Support

==================================================
When responding to posts, please "Reply to Group" via your newsreader so
that others may learn and benefit from your issue.
==================================================

This posting is provided "AS IS" with no warranties, and confers no rights.