OOM error with lot of available process memory

[Beorne]

I'm using WindowsCE 5.0 with CF 2.0

I'm trying to detect the available memory so to detect when the memory
usage is low.
After a good search time I've found that the good indicator of
available memory is tha dwAvailVirtual field of the memoryStatus
structure, that considers the process memory and not the whole memory.

I've built a test application that at every cycle allocates an array
32K more than the previous (in practice it allocates the allocable
minimum, 64K).

for (int i = 1; i < 100000; i ++)
{
GC.Collect(); // probably useless

byte[] array = new byte[1024 * i * 32];

// probably useless
for (int j = 0; j < array.Length; j++)
array[j] = (byte)0;

Util.GlobalMemoryStatusCE(ref ms); //wrapper for
GlobalMemoryStatus
str =
"GC.GetTotalMemory = " + (int)(GC.GetTotalMemory
(false) / 1024) + "\n" +
"ms.MemoryLoad = " + ms.dwMemoryLoad + "\n" +
"ms.TotalPhys = " + (int)(ms.dwTotalPhys / 1024) +
"\n" +
"ms.AvailPhys = " + (int)(ms.dwAvailPhys / 1024) +
"\n" +
"ms.dwTotalVirtual = " + (int)(ms.dwTotalVirtual /
1024) + "\n" +
"ms.dwAvailVirtual = " + (int)(ms.dwAvailVirtual /
1024) + "\n";

Console.WriteLine(i + "\n" + str.ToString());
}

The problem is that the application goes in out of memory when there
are 9 Mb of allocable memory left ...
Last printing:

303
GC.GetTotalMemory = 9699
ms.MemoryLoad = 48
ms.TotalPhys = 78552
ms.AvailPhys = 41356
ms.dwTotalVirtual = 32768
ms.dwAvailVirtual = 9792 <----

Why?

Thanks.

 

[Chris Tacke, MVP]

Because you're out of virtual memory. Your app does not have access to all
of the device memory, and if you're allocating from the stack (as opposed to
the GC heap) then it comes from process memory.

There are plenty of online resources that explain CE's memory model as well
as that of the CF.


--

Chris Tacke, Embedded MVP
OpenNETCF Consulting
Giving back to the embedded community
http://community.OpenNETCF.com

 

[Beorne]

Yes, I have understood this, in fact from the example I'm showing you
that I when the program does oom I have yet 9 Mb free of Virtual
Memory (that belongs to the 32Mb of process memory).
I'm not considering the 41Mb of physical memory that I know are not
the issue.

Because you're out of virtual memory.  Your app does not have access toall
of the device memory, and if you're allocating from the stack (as opposedto
the GC heap) then it comes from process memory.

There are plenty of online resources that explain CE's memory model as well
as that of the CF.

--

Chris Tacke, Embedded MVP
OpenNETCF Consulting
Giving back to the embedded communityhttp://community.OpenNETCF.com

I'm using WindowsCE 5.0 with CF 2.0

I'm trying to detect the available memory so to detect when the memory
usage is low.
After a good search time I've found that the good indicator of
available memory is tha dwAvailVirtual field of the memoryStatus
structure, that considers the process memory and not the whole memory.

I've built a test application that at every cycle allocates an array
32K more than the previous (in practice it allocates the allocable
minimum, 64K).

           for (int i = 1; i < 100000; i ++)
           {
               GC.Collect(); // probably useless

               byte[] array = new byte[1024 * i * 32];

               // probably useless
               for (int j = 0; j < array.Length; j++)
                   array[j] = (byte)0;

               Util.GlobalMemoryStatusCE(ref ms); //wrapper for
GlobalMemoryStatus
               str =
                   "GC.GetTotalMemory = " + (int)(GC.GetTotalMemory
(false) / 1024) + "\n" +
                   "ms.MemoryLoad = " + ms.dwMemoryLoad + "\n" +
                   "ms.TotalPhys = " + (int)(ms.dwTotalPhys / 1024) +
"\n" +
                   "ms.AvailPhys = " + (int)(ms.dwAvailPhys / 1024) +
"\n" +
                   "ms.dwTotalVirtual = " + (int)(ms.dwTotalVirtual /
1024) + "\n" +
                   "ms.dwAvailVirtual = " + (int)(ms.dwAvailVirtual /
1024) + "\n";

               Console.WriteLine(i + "\n" + str.ToString());
           }

The problem is that the application goes in out of memory when there
are 9 Mb of allocable memory left ...
Last printing:

303
GC.GetTotalMemory = 9699
ms.MemoryLoad = 48
ms.TotalPhys = 78552
ms.AvailPhys = 41356
ms.dwTotalVirtual = 32768
ms.dwAvailVirtual = 9792 <----

Thanks.

 

[Chris Tacke, eMVP]

OK, so what's the question then?

if your last output was 303, then the next attempt to allocate will be

1024 * 304 * 32 == 9,961,472 bytes.

You have 9.7MB free and last I checked 9.9 (your attempt) is greater than
9.7 (what's available), so not surprisingly you get an error.

Your code doesn't output the failure values, plus what you're doing is going
to require contiguous memory, and you certainly can't assume that the number
reported as available is all contiguous.


--

Chris Tacke, Embedded MVP
OpenNETCF Consulting
Giving back to the embedded community
http://community.OpenNETCF.com


Yes, I have understood this, in fact from the example I'm showing you
that I when the program does oom I have yet 9 Mb free of Virtual
Memory (that belongs to the 32Mb of process memory).
I'm not considering the 41Mb of physical memory that I know are not
the issue.

Because you're out of virtual memory. Your app does not have access to all
of the device memory, and if you're allocating from the stack (as opposed
to
the GC heap) then it comes from process memory.

There are plenty of online resources that explain CE's memory model as
well
as that of the CF.

--

Chris Tacke, Embedded MVP
OpenNETCF Consulting
Giving back to the embedded communityhttp://community.OpenNETCF.com

I'm using WindowsCE 5.0 with CF 2.0

I'm trying to detect the available memory so to detect when the memory
usage is low.
After a good search time I've found that the good indicator of
available memory is tha dwAvailVirtual field of the memoryStatus
structure, that considers the process memory and not the whole memory.

I've built a test application that at every cycle allocates an array
32K more than the previous (in practice it allocates the allocable
minimum, 64K).

for (int i = 1; i < 100000; i ++)
{
GC.Collect(); // probably useless

byte[] array = new byte[1024 * i * 32];

// probably useless
for (int j = 0; j < array.Length; j++)
array[j] = (byte)0;

Util.GlobalMemoryStatusCE(ref ms); //wrapper for
GlobalMemoryStatus
str =
"GC.GetTotalMemory = " + (int)(GC.GetTotalMemory
(false) / 1024) + "\n" +
"ms.MemoryLoad = " + ms.dwMemoryLoad + "\n" +
"ms.TotalPhys = " + (int)(ms.dwTotalPhys / 1024) +
"\n" +
"ms.AvailPhys = " + (int)(ms.dwAvailPhys / 1024) +
"\n" +
"ms.dwTotalVirtual = " + (int)(ms.dwTotalVirtual /
1024) + "\n" +
"ms.dwAvailVirtual = " + (int)(ms.dwAvailVirtual /
1024) + "\n";

Console.WriteLine(i + "\n" + str.ToString());
}

The problem is that the application goes in out of memory when there
are 9 Mb of allocable memory left ...
Last printing:

303
GC.GetTotalMemory = 9699
ms.MemoryLoad = 48
ms.TotalPhys = 78552
ms.AvailPhys = 41356
ms.dwTotalVirtual = 32768
ms.dwAvailVirtual = 9792 <----

Thanks.

 

[Beorne]

The question is:
why the gc does not collect the previous out of scope array before
trying to commit the remaining available virtual memory?

OK, so what's the question then?

if your last output was 303, then the next attempt to allocate will be

1024 * 304 * 32 == 9,961,472 bytes.

You have 9.7MB free and last I checked 9.9 (your attempt) is greater than
9.7 (what's available), so not surprisingly you get an error.

Your code doesn't output the failure values, plus what you're doing is going
to require contiguous memory, and you certainly can't assume that the number
reported as available is all contiguous.

--

Chris Tacke, Embedded MVP
OpenNETCF Consulting
Giving back to the embedded communityhttp://community.OpenNETCF.com


Yes, I have understood this, in fact from the example I'm showing you
that I when the program does oom I have yet 9 Mb free of Virtual
Memory (that belongs to the 32Mb of process memory).
I'm not considering the 41Mb of physical memory that I know are not
the issue.

Because you're out of virtual memory. Your app does not have access to all
of the device memory, and if you're allocating from the stack (as opposed
to
the GC heap) then it comes from process memory.

There are plenty of online resources that explain CE's memory model as
well
as that of the CF.

Chris Tacke, Embedded MVP
OpenNETCF Consulting
Giving back to the embedded communityhttp://community.OpenNETCF.com

news:[email protected]...

I'm using WindowsCE 5.0 with CF 2.0
I'm trying to detect the available memory so to detect when the memory
usage is low.
After a good search time I've found that the good indicator of
available memory is tha dwAvailVirtual field of the memoryStatus
structure, that considers the process memory and not the whole memory.
I've built a test application that at every cycle allocates an array
32K more than the previous (in practice it allocates the allocable
minimum, 64K).
for (int i = 1; i < 100000; i ++)
{
GC.Collect(); // probably useless
byte[] array = new byte[1024 * i * 32];
// probably useless
for (int j = 0; j < array.Length; j++)
array[j] = (byte)0;
Util.GlobalMemoryStatusCE(ref ms); //wrapper for
GlobalMemoryStatus
str =
"GC.GetTotalMemory = " + (int)(GC.GetTotalMemory
(false) / 1024) + "\n" +
"ms.MemoryLoad = " + ms.dwMemoryLoad + "\n" +
"ms.TotalPhys = " + (int)(ms.dwTotalPhys / 1024) +
"\n" +
"ms.AvailPhys = " + (int)(ms.dwAvailPhys / 1024) +
"\n" +
"ms.dwTotalVirtual = " + (int)(ms.dwTotalVirtual /
1024) + "\n" +
"ms.dwAvailVirtual = " + (int)(ms.dwAvailVirtual /
1024) + "\n";
Console.WriteLine(i + "\n" + str.ToString());
}
The problem is that the application goes in out of memory when there
are 9 Mb of allocable memory left ...
Last printing:
303
GC.GetTotalMemory = 9699
ms.MemoryLoad = 48
ms.TotalPhys = 78552
ms.AvailPhys = 41356
ms.dwTotalVirtual = 32768
ms.dwAvailVirtual = 9792 <----
Why?
Thanks.

 

[Chris Tacke, eMVP]

How much VM is available before you make any of these allocations?
Have you tried to catch the OOM, call Collect in the handler and
re-allocate?
Why does this even matter? Unless you have an app that does nothing but
generate these gigantic arrays, I don't see the entire point of this
exercise. You must be trying to solve some real problem, right? What
problem is that? If you need a 10MB contiguous array for something, there
are probably better ways to allocate.


--

Chris Tacke, Embedded MVP
OpenNETCF Consulting
Giving back to the embedded community
http://community.OpenNETCF.com

The question is:
why the gc does not collect the previous out of scope array before
trying to commit the remaining available virtual memory?

OK, so what's the question then?

if your last output was 303, then the next attempt to allocate will be

1024 * 304 * 32 == 9,961,472 bytes.

You have 9.7MB free and last I checked 9.9 (your attempt) is greater than
9.7 (what's available), so not surprisingly you get an error.

Your code doesn't output the failure values, plus what you're doing is
going
to require contiguous memory, and you certainly can't assume that the
number
reported as available is all contiguous.

--

Chris Tacke, Embedded MVP
OpenNETCF Consulting
Giving back to the embedded communityhttp://community.OpenNETCF.com


Yes, I have understood this, in fact from the example I'm showing you
that I when the program does oom I have yet 9 Mb free of Virtual
Memory (that belongs to the 32Mb of process memory).
I'm not considering the 41Mb of physical memory that I know are not
the issue.

Because you're out of virtual memory. Your app does not have access to
all
of the device memory, and if you're allocating from the stack (as
opposed
to
the GC heap) then it comes from process memory.

There are plenty of online resources that explain CE's memory model as
well
as that of the CF.

Chris Tacke, Embedded MVP
OpenNETCF Consulting
Giving back to the embedded communityhttp://community.OpenNETCF.com

I'm using WindowsCE 5.0 with CF 2.0

I'm trying to detect the available memory so to detect when the
memory
usage is low.
After a good search time I've found that the good indicator of
available memory is tha dwAvailVirtual field of the memoryStatus
structure, that considers the process memory and not the whole
memory.

I've built a test application that at every cycle allocates an array
32K more than the previous (in practice it allocates the allocable
minimum, 64K).

for (int i = 1; i < 100000; i ++)
{
GC.Collect(); // probably useless

byte[] array = new byte[1024 * i * 32];

// probably useless
for (int j = 0; j < array.Length; j++)
array[j] = (byte)0;

Util.GlobalMemoryStatusCE(ref ms); //wrapper for
GlobalMemoryStatus
str =
"GC.GetTotalMemory = " + (int)(GC.GetTotalMemory
(false) / 1024) + "\n" +
"ms.MemoryLoad = " + ms.dwMemoryLoad + "\n" +
"ms.TotalPhys = " + (int)(ms.dwTotalPhys / 1024) +
"\n" +
"ms.AvailPhys = " + (int)(ms.dwAvailPhys / 1024) +
"\n" +
"ms.dwTotalVirtual = " + (int)(ms.dwTotalVirtual /
1024) + "\n" +
"ms.dwAvailVirtual = " + (int)(ms.dwAvailVirtual /
1024) + "\n";

Console.WriteLine(i + "\n" + str.ToString());
}

The problem is that the application goes in out of memory when there
are 9 Mb of allocable memory left ...
Last printing:

303
GC.GetTotalMemory = 9699
ms.MemoryLoad = 48
ms.TotalPhys = 78552
ms.AvailPhys = 41356
ms.dwTotalVirtual = 32768
ms.dwAvailVirtual = 9792 <----

Thanks.

 

[Beorne]

You are right, allocating every cycle a byte[1024 * 32] I don't
claim contiguous memory and I can reach a wonderful
ms.dwAvailVirtual = 0
before going oom

Thank you!

How much VM is available before you make any of these allocations?
Have you tried to catch the OOM, call Collect in the handler and
re-allocate?
Why does this even matter?  Unless you have an app that does nothing but
generate these gigantic arrays, I don't see the entire point of this
exercise.  You must be trying to solve some real problem, right?  What
problem is that?  If you need a 10MB contiguous array for something, there
are probably better ways to allocate.

--

Chris Tacke, Embedded MVP
OpenNETCF Consulting
Giving back to the embedded communityhttp://community.OpenNETCF.com

The question is:
why the gc does not collect the previous out of scope array before
trying to commit the remaining available virtual memory?

OK, so what's the question then?
if your last output was 303, then the next attempt to allocate will be
1024 * 304 * 32 == 9,961,472 bytes.
You have 9.7MB free and last I checked 9.9 (your attempt) is greater than
9.7 (what's available), so not surprisingly you get an error.
Your code doesn't output the failure values, plus what you're doing is
going
to require contiguous memory, and you certainly can't assume that the
number
reported as available is all contiguous.
--
Chris Tacke, Embedded MVP
OpenNETCF Consulting
Giving back to the embedded communityhttp://community.OpenNETCF.com
Yes, I have understood this, in fact from the example I'm showing you
that I when the program does oom I have yet 9 Mb free of Virtual
Memory (that belongs to the 32Mb of process memory).
I'm not considering the 41Mb of physical memory that I know are not
the issue.
On 8 Apr, 14:37, "Chris Tacke, MVP" <ctacke.at.opennetcf.dot.com>
wrote:
Because you're out of virtual memory. Your app does not have access to
all
of the device memory, and if you're allocating from the stack (as
opposed
to
the GC heap) then it comes from process memory.
There are plenty of online resources that explain CE's memory model as
well
as that of the CF.
--
Chris Tacke, Embedded MVP
OpenNETCF Consulting
Giving back to the embedded communityhttp://community.OpenNETCF.com

I'm using WindowsCE 5.0 with CF 2.0
I'm trying to detect the available memory so to detect when the
memory
usage is low.
After a good search time I've found that the good indicator of
available memory is tha dwAvailVirtual field of the memoryStatus
structure, that considers the process memory and not the whole
memory.
I've built a test application that at every cycle allocates an array
32K more than the previous (in practice it allocates the allocable
minimum, 64K).
for (int i = 1; i < 100000; i ++)
{
GC.Collect(); // probably useless
byte[] array = new byte[1024 * i * 32];
// probably useless
for (int j = 0; j < array.Length; j++)
array[j] = (byte)0;
Util.GlobalMemoryStatusCE(ref ms); //wrapper for
GlobalMemoryStatus
str =
"GC.GetTotalMemory = " + (int)(GC.GetTotalMemory
(false) / 1024) + "\n" +
"ms.MemoryLoad = " + ms.dwMemoryLoad + "\n" +
"ms.TotalPhys = " + (int)(ms.dwTotalPhys / 1024) +
"\n" +
"ms.AvailPhys = " + (int)(ms.dwAvailPhys / 1024) +
"\n" +
"ms.dwTotalVirtual = " + (int)(ms.dwTotalVirtual /
1024) + "\n" +
"ms.dwAvailVirtual = " + (int)(ms.dwAvailVirtual /
1024) + "\n";
Console.WriteLine(i + "\n" + str.ToString());
}
The problem is that the application goes in out of memory when there
are 9 Mb of allocable memory left ...
Last printing:
303
GC.GetTotalMemory = 9699
ms.MemoryLoad = 48
ms.TotalPhys = 78552
ms.AvailPhys = 41356
ms.dwTotalVirtual = 32768
ms.dwAvailVirtual = 9792 <----
Why?
Thanks.