Re: .NET Framework Can't use Available Memory

[Alec Seward]

Jim,

I have managed to squeeze out 2.4 Gb to one process in NET v1.1, on a 4Gb
computer running WinXP Pro.

1. Add the /3GB switch in boot.ini (as you did)
2. After building you program run the linker (located in e.g. C:\Program
Files\Microsoft Visual Studio .NET 2003\Vc7\bin)
link -edit -LARGEADDRESSAWARE target.exe
where "target.exe" is your application executable

Now run the executable and you should be able to allocate up to 2.4Gb to the
process.
Why it stops there and don't reach 3Gb, I don't know.
An unmanaged process can of course access all of the available 3Gb of user
memory in Windows).

Does anyone know if it is possible to do this in .NET/CLR as well?


Best

Alec Seward

"Memory limitations, memory limitations...why are there so many of them?"

----- Original Message -----
From: "Jim Snape" <[email protected]>
Newsgroups: microsoft.public.dotnet.framework
Sent: July 15, 2003 14:31
Subject: .NET Framework Can't use Available Memory

 

[Alec Seward]

Scott, thanks for your answer.

I cannot argue with your reasoning, but does it really give the whole
picture??

When running a C program (WinXp with 4Gb 3GB swittch), allocation (in small
chunks) can continue until the whole process consumes 3Gb (quite on the byte
actually).
When switching to C#, doing the same thing, it is impossible (at least for
me) to get past the 2.4Gb barrier for the whole process. Note that this is
the total memory for the process (as in the C case), not just what's been
allocated.

600 MB is missing here!! Running a minimal console app in C# lowers the
ceiling of available RAM compared to unmanaged code by 600 MBytes . And you
are saying that all this wasted memory is is eaten up by loaded NET DLL's
and the GC ??!? Many computers does not even have that much RAM.

Please tell me there is a workaround. Does really a minimalistic ten line C#
program (that only allocates integers) really require 600 Mb extra of loaded
DLL's, or....is it instead so "complicated" that the GC must reserve all
those bytes just in case.

If the GC is the devil here, I rather see it reserve less memory (and take
the risk of not be able to garbage collect my simple int allocations),
instead of my app crashing with an OutOfMemoryException with 600 Mb still
unused.

I missing something here?


Regards
/Alec Seward

This behavior you are seeing is normal by design.

The /3gb switch and LARGEADDRESSAWARE link flag cause the OS to allow a
process to use 3gb of addressable user space.

(history for any other readers of this thread)
Each "normal" process has 4gb of address space (consider a 32-bit pointer).
There are extentions that allow you slide a window through more memory,
but fundamenatlly you cannot escape the fact that common x86 machines only
support 32-bit addresses (therefore 4gb). Normally the OS uses the top
half of memory (virtual address space) for system space. (i.e. thats where
the the "kernel" lives). If you pass /3GB in the boot.ini as a flag to the
kernel, it will an extra check during the loading of a process. This extra
check will look for the LARGEADDRESSAWARE flag (settable by linker option
or with imagecfg). If that flag is set, the OS will erect a process with
3gb user space, and 1gb system space. However, through that 3gb of system
space, there are already quite a few little modules sprinkled about.
If you review the allocations that already exist on process creation,
you'll see that there is quite a bit thats already allocated. (again this
is va space thats allocated.) A simple "lm" after loading a small native
app in ntsd (say "ntsd calc.exe") shows:
0:000> lm
start end module name
01000000 0101f000 calc (deferred)
77290000 772d9000 SHLWAPI (deferred)
77380000 77b5d000 SHELL32 (deferred)
77ba0000 77bf4000 msvcrt (deferred)
77c00000 77c44000 GDI32 (deferred)
77c50000 77cf4000 RPCRT4 (deferred)
77d00000 77d8f000 USER32 (deferred)
77da0000 77e30000 ADVAPI32 (deferred)
77e40000 77f34000 kernel32 (deferred)
77f40000 77ffa000 ntdll (pdb symbols)

(yes.. thats a 2gb machine... i dont have my 3gb machine handy right now)
Meaning that the largest linear region of memory inside 2gb we could
allocate at this point would be 0x76271000 or 1.9 gigs.
When you have a more complicated app in memory, (say something running
managed code) you have many more dll's loaded at various places in memory.

That said, the gc is only able to allocate from the pool of free memory,
and have a few hundred megs of virtual address space consumed when an app
is running is not uncommong. (thats va space. so, it just means that its
atleast reserved, and maybe not actually allocated). On some Oses (depends
on what all is being loaded into a process) your ability to allocate
certain amounts of memory will vary. So, for instance running inside a
debugger will lower the amount you'll be able to allocate.

I'd expect that in general you'll have better luck trying to allocate
smaller chunks. But, dont expect to be able to get the full 3gb (with /3gb
enabled) or the full 2gb with normal 2gb enabled.

-scott

--------------------

From: "Alec Seward" <[email protected]>
Subject: Re: .NET Framework Can't use Available Memory
Date: Thu, 17 Jul 2003 10:57:01 +0200

Jim,

I have managed to squeeze out 2.4 Gb to one process in NET v1.1, on a 4Gb
computer running WinXP Pro.

1. Add the /3GB switch in boot.ini (as you did)
2. After building you program run the linker (located in e.g. C:\Program
Files\Microsoft Visual Studio .NET 2003\Vc7\bin)
link -edit -LARGEADDRESSAWARE target.exe
where "target.exe" is your application executable

Now run the executable and you should be able to allocate up to 2.4Gb to the
process.
Why it stops there and don't reach 3Gb, I don't know.
An unmanaged process can of course access all of the available 3Gb of user
memory in Windows).

Does anyone know if it is possible to do this in .NET/CLR as well?


Best

Alec Seward

"Memory limitations, memory limitations...why are there so many of them?"

----- Original Message -----
From: "Jim Snape" <[email protected]>
Newsgroups: microsoft.public.dotnet.framework
Sent: July 15, 2003 14:31
Subject: .NET Framework Can't use Available Memory

I need to cache large amounts of data for fast lookup, hence I've been
running some tests on various operating systems (XP, Win2k Adv etc) and it
seems that no matter how much physical memory is available a .NET framework
application can not use more that 1.2GB. I have even tried with a 4GB
machine using the 3GB boot time switch.

Is there some hard limit built into the framework? I can't believe it would
be handicapped in this manor. Is there any way of circumventing the limit?

Thanks in advance,
Jim

p.s. For reference, the code I used to test the limit is:

using System;

namespace MemAlloc
{
class MemBlock
{
private MemBlock next;
private byte[] data = new byte[1024*64];

const int OneMeg = 1024 * 1024;

static void Main(string[] args)
{
try
{
MemBlock first = new MemBlock();

while (true)
{
MemBlock next = new MemBlock();
next.next = first;
first = next;

Console.WriteLine("MEM: {0} MB",
GC.GetTotalMemory(false)/OneMeg);
}
}
catch (Exception ex)
{
Console.WriteLine("{0}: {1}", ex.Source, ex.Message);
}
}
}
}

--

This posting is provided "AS IS" with no warranties, and confers no rights.
Use of included script samples are subject to the terms specified at
http://www.microsoft.com/info/cpyright.htm

Note: For the benefit of the community-at-large, all responses to this
message are best directed to the newsgroup/thread from which they
originated.

 

[Alec Seward]

Scott,

here are the last output lines of the C and the C# programs before memory
allocation failure and program crash (C: malloc failed, C#:
OutOfMemoryException)
All apps were compiled using VStudio 2003 EA.

C used the /LARGEADDRESSAWARE linker option.
In the C# case the app was relinked
link -edit -LARGEADDRESSAWARE Memtest.exe (why isn't there a linker options
in VStudio for C# projects?)

As you can see, C# version falls short of 600Mb (with LARGEADDRESSAWARE).
I should add that I've tried several other code snippets with the same
results.

What do you think... what is causing the difference??

Program output before crash and the both code snippets attached below.....
Scott, could you please also describe your test setup?

/Alec Seward

---- Native C, no /LARGEADDRESSAWARE ----
1996000 Kb allocated
process size according to taskManager: 2.0Gb

---- Native C, with /LARGEADDRESSAWARE ----
2998000 Kb allocated
process size according to taskManager: : 3.0Gb

---- C#, no /LARGEADDRESSAWARE ----
1455000 Kb allocated
process size according to taskManager: : 1.5Gb

---- C#, with /LARGEADDRESSAWARE ----
2328000 Kb allocated
process size according to taskManager: : 2.4Gb


---------- Native C version -----------------------
#include <stdio.h>
#include <stdlib.h>
void TestMemoryLimit() {
int *x;
for (int i = 0; i < 10000000; i ++) {
x = (int*) malloc(256 * sizeof(int)); //allocate 1K array
if (x == NULL) {
printf("Malloc failed\n");
break;
}
if (i % 1000 == 0) {
printf("%d Kb allocated\n", i);
}
}
}

void main() {
printf("Hello\n");
TestMemoryLimit();
}

---------- Managed C# version -----------------------
using System;
public class MemTest {

static void TestMemoryLimit() {
int[][] x = new int[10000000][];
for (int i = 0; i < x.Length; i ++) {
x = new int[256]; //allocate 1K array
if (i % 1000 == 0) {
Console.WriteLine(i + " Kb allocated");
}
}
}
public static void Main() {
TestMemoryLimit();
}

}

---------- End -----------------------

600megs is quite a big difference. I dont see anything that bad on my
machines.
Can you drop me the exact apps you are comparing?

in using a small variant of whats shown below, i see a managed version that
eats 1mb at a time, peak out at 1755mb.
A native version of bascially the same thing is able to allocate about
1822mb of native 1mb chunks.

If I descrease the block size down to 64k i can allocate ~1820mb in the
managed process.
And the same attempt in native yields a peak of about 1900mb.

On one of my 3gb boxes the most i was able to allocate was 2.4gb managed,
and 2.5gb native. (in 1mb chunks).

The peak that i can allocate native(and managed) varies quite a bit from
machine to machine.

Its important to remeber a few things when comparing managed with native.
1. GC.GetTotalMemory is reporting the number of user visable bytes in
the gc allocated memory. (each will have a small overhead. so if you have
a million objects, expect a few megs of overhead. etc.)
2. The space "consumed" in memory by dll's is only the VA space. What
this means is that address space(a chunk of memory) can really be in the
three fundamental states.
A. Allocated and commited - Meaning that you have done a new and
that the memory can be directly written. It might consume physical ram, or
it might consume space on disk in a swap file.
B. Allocated and not commited - When a dll is mapped into memory for
instance, address space is reserved for it (depending on the situation, the
dll might need a big linear region.), but since we didnt load the whole
dll, we never need to commit the space.
C. Free - this memory is open to be allocated. Though it might be
fragmented, and we might only have little 64k chunks all over the place
that are free.

The GC is likely not the reason for the 600meg diff you are seeing. Drop
me the little snippets that make up the example, and i'd love to take a
detailed look and let you know whats going on.

thanks!
-Scott


--------------------

From: "Alec Seward" <[email protected]>
Subject: Re: .NET Framework Can't use Available Memory
Date: Sat, 19 Jul 2003 00:03:44 +0200

Scott, thanks for your answer.

I cannot argue with your reasoning, but does it really give the whole
picture??

When running a C program (WinXp with 4Gb 3GB swittch), allocation (in small
chunks) can continue until the whole process consumes 3Gb (quite on the byte
actually).
When switching to C#, doing the same thing, it is impossible (at least for
me) to get past the 2.4Gb barrier for the whole process. Note that this is
the total memory for the process (as in the C case), not just what's been
allocated.

600 MB is missing here!! Running a minimal console app in C# lowers the
ceiling of available RAM compared to unmanaged code by 600 MBytes . And you
are saying that all this wasted memory is is eaten up by loaded NET DLL's
and the GC ??!? Many computers does not even have that much RAM.

Please tell me there is a workaround. Does really a minimalistic ten line C#
program (that only allocates integers) really require 600 Mb extra of loaded
DLL's, or....is it instead so "complicated" that the GC must reserve all
those bytes just in case.

If the GC is the devil here, I rather see it reserve less memory (and take
the risk of not be able to garbage collect my simple int allocations),
instead of my app crashing with an OutOfMemoryException with 600 Mb still
unused.

I missing something here?


Regards
/Alec Seward

This behavior you are seeing is normal by design.

The /3gb switch and LARGEADDRESSAWARE link flag cause the OS to allow a
process to use 3gb of addressable user space.

(history for any other readers of this thread)
Each "normal" process has 4gb of address space (consider a 32-bit pointer).
There are extentions that allow you slide a window through more memory,
but fundamenatlly you cannot escape the fact that common x86 machines only
support 32-bit addresses (therefore 4gb). Normally the OS uses the top
half of memory (virtual address space) for system space. (i.e. thats where
the the "kernel" lives). If you pass /3GB in the boot.ini as a flag to the
kernel, it will an extra check during the loading of a process. This extra
check will look for the LARGEADDRESSAWARE flag (settable by linker option
or with imagecfg). If that flag is set, the OS will erect a process with
3gb user space, and 1gb system space. However, through that 3gb of system
space, there are already quite a few little modules sprinkled about.
If you review the allocations that already exist on process creation,
you'll see that there is quite a bit thats already allocated. (again this
is va space thats allocated.) A simple "lm" after loading a small native
app in ntsd (say "ntsd calc.exe") shows:
0:000> lm
start end module name
01000000 0101f000 calc (deferred)
77290000 772d9000 SHLWAPI (deferred)
77380000 77b5d000 SHELL32 (deferred)
77ba0000 77bf4000 msvcrt (deferred)
77c00000 77c44000 GDI32 (deferred)
77c50000 77cf4000 RPCRT4 (deferred)
77d00000 77d8f000 USER32 (deferred)
77da0000 77e30000 ADVAPI32 (deferred)
77e40000 77f34000 kernel32 (deferred)
77f40000 77ffa000 ntdll (pdb symbols)

(yes.. thats a 2gb machine... i dont have my 3gb machine handy right now)
Meaning that the largest linear region of memory inside 2gb we could
allocate at this point would be 0x76271000 or 1.9 gigs.
When you have a more complicated app in memory, (say something running
managed code) you have many more dll's loaded at various places in memory.

That said, the gc is only able to allocate from the pool of free memory,
and have a few hundred megs of virtual address space consumed when an app
is running is not uncommong. (thats va space. so, it just means that its
atleast reserved, and maybe not actually allocated). On some Oses (depends
on what all is being loaded into a process) your ability to allocate
certain amounts of memory will vary. So, for instance running inside a
debugger will lower the amount you'll be able to allocate.

I'd expect that in general you'll have better luck trying to allocate
smaller chunks. But, dont expect to be able to get the full 3gb (with /3gb
enabled) or the full 2gb with normal 2gb enabled.

-scott

--------------------
From: "Alec Seward" <[email protected]>
Subject: Re: .NET Framework Can't use Available Memory
Date: Thu, 17 Jul 2003 10:57:01 +0200

Jim,

I have managed to squeeze out 2.4 Gb to one process in NET v1.1, on a 4Gb
computer running WinXP Pro.

1. Add the /3GB switch in boot.ini (as you did)
2. After building you program run the linker (located in e.g. C:\Program
Files\Microsoft Visual Studio .NET 2003\Vc7\bin)
link -edit -LARGEADDRESSAWARE target.exe
where "target.exe" is your application executable

Now run the executable and you should be able to allocate up to 2.4Gb to
the
process.
Why it stops there and don't reach 3Gb, I don't know.
An unmanaged process can of course access all of the available 3Gb of user
memory in Windows).

Does anyone know if it is possible to do this in .NET/CLR as well?


Best

Alec Seward

"Memory limitations, memory limitations...why are there so many of them?"

----- Original Message -----
From: "Jim Snape" <[email protected]>
Newsgroups: microsoft.public.dotnet.framework
Sent: July 15, 2003 14:31
Subject: .NET Framework Can't use Available Memory


I need to cache large amounts of data for fast lookup, hence I've been
running some tests on various operating systems (XP, Win2k Adv etc) and
it
seems that no matter how much physical memory is available a .NET
framework
application can not use more that 1.2GB. I have even tried with a 4GB
machine using the 3GB boot time switch.

Is there some hard limit built into the framework? I can't believe it
would
be handicapped in this manor. Is there any way of circumventing the
limit?

Thanks in advance,
Jim

p.s. For reference, the code I used to test the limit is:

using System;

namespace MemAlloc
{
class MemBlock
{
private MemBlock next;
private byte[] data = new byte[1024*64];

const int OneMeg = 1024 * 1024;

static void Main(string[] args)
{
try
{
MemBlock first = new MemBlock();

while (true)
{
MemBlock next = new MemBlock();
next.next = first;
first = next;

Console.WriteLine("MEM: {0} MB",
GC.GetTotalMemory(false)/OneMeg);
}
}
catch (Exception ex)
{
Console.WriteLine("{0}: {1}", ex.Source, ex.Message);
}
}
}
}







--

This posting is provided "AS IS" with no warranties, and confers no rights.
Use of included script samples are subject to the terms specified at
http://www.microsoft.com/info/cpyright.htm

Note: For the benefit of the community-at-large, all responses to this
message are best directed to the newsgroup/thread from which they
originated.

--

This posting is provided "AS IS" with no warranties, and confers no rights.
Use of included script samples are subject to the terms specified at
http://www.microsoft.com/info/cpyright.htm

Note: For the benefit of the community-at-large, all responses to this
message are best directed to the newsgroup/thread from which they
originated.