Re: .NET Framework Can't use Available Memory

Jim,

I have managed to squeeze out 2.4 Gb to one process in NET v1.1, on a 4Gb
computer running WinXP Pro.

1. Add the /3GB switch in boot.ini (as you did)
2. After building you program run the linker (located in e.g. C:\Program
Files\Microsoft Visual Studio .NET 2003\Vc7\bin)
link -edit -LARGEADDRESSAWARE target.exe
where "target.exe" is your application executable

Now run the executable and you should be able to allocate up to 2.4Gb to the
process.
Why it stops there and don't reach 3Gb, I don't know.
An unmanaged process can of course access all of the available 3Gb of user
memory in Windows).

Does anyone know if it is possible to do this in .NET/CLR as well?


Best

Alec Seward

"Memory limitations, memory limitations...why are there so many of them?"

----- Original Message -----
From: "Jim Snape" <jsjunk_2002.deletethiss@yahoo.co.uk>
Newsgroups: microsoft.public.dotnet.framework
Sent: July 15, 2003 14:31
Subject: .NET Framework Can't use Available Memory


> I need to cache large amounts of data for fast lookup, hence I've been
> running some tests on various operating systems (XP, Win2k Adv etc) and it
> seems that no matter how much physical memory is available a .NET
framework
> application can not use more that 1.2GB. I have even tried with a 4GB
> machine using the 3GB boot time switch.
>
> Is there some hard limit built into the framework? I can't believe it
would
> be handicapped in this manor. Is there any way of circumventing the limit?
>
> Thanks in advance,
> Jim
>
> p.s. For reference, the code I used to test the limit is:
>
> using System;
>
> namespace MemAlloc
> {
> class MemBlock
> {
> private MemBlock next;
> private byte[] data = new byte[1024*64];
>
> const int OneMeg = 1024 * 1024;
>
> static void Main(string[] args)
> {
> try
> {
> MemBlock first = new MemBlock();
>
> while (true)
> {
> MemBlock next = new MemBlock();
> next.next = first;
> first = next;
>
> Console.WriteLine("MEM: {0} MB",
> GC.GetTotalMemory(false)/OneMeg);
> }
> }
> catch (Exception ex)
> {
> Console.WriteLine("{0}: {1}", ex.Source, ex.Message);
> }
> }
> }
> }
>
>

Scott, thanks for your answer.

I cannot argue with your reasoning, but does it really give the whole
picture??

When running a C program (WinXp with 4Gb 3GB swittch), allocation (in small
chunks) can continue until the whole process consumes 3Gb (quite on the byte
actually).
When switching to C#, doing the same thing, it is impossible (at least for
me) to get past the 2.4Gb barrier for the whole process. Note that this is
the total memory for the process (as in the C case), not just what's been
allocated.

600 MB is missing here!! Running a minimal console app in C# lowers the
ceiling of available RAM compared to unmanaged code by 600 MBytes . And you
are saying that all this wasted memory is is eaten up by loaded NET DLL's
and the GC ??!? Many computers does not even have that much RAM.

Please tell me there is a workaround. Does really a minimalistic ten line C#
program (that only allocates integers) really require 600 Mb extra of loaded
DLL's, or....is it instead so "complicated" that the GC must reserve all
those bytes just in case.

If the GC is the devil here, I rather see it reserve less memory (and take
the risk of not be able to garbage collect my simple int allocations),
instead of my app crashing with an OutOfMemoryException with 600 Mb still
unused.

I missing something here?


Regards
/Alec Seward




"Scott Wadsworth [MSFT]" <bwadswor@online.microsoft.com> wrote in message
news:lSBjK9WTDHA.1924@cpmsftngxa06.phx.gbl...
> This behavior you are seeing is normal by design.
>
> The /3gb switch and LARGEADDRESSAWARE link flag cause the OS to allow a
> process to use 3gb of addressable user space.
>
> (history for any other readers of this thread)
> Each "normal" process has 4gb of address space (consider a 32-bit
pointer).
> There are extentions that allow you slide a window through more memory,
> but fundamenatlly you cannot escape the fact that common x86 machines only
> support 32-bit addresses (therefore 4gb). Normally the OS uses the top
> half of memory (virtual address space) for system space. (i.e. thats
where
> the the "kernel" lives). If you pass /3GB in the boot.ini as a flag to
the
> kernel, it will an extra check during the loading of a process. This
extra
> check will look for the LARGEADDRESSAWARE flag (settable by linker option
> or with imagecfg). If that flag is set, the OS will erect a process with
> 3gb user space, and 1gb system space. However, through that 3gb of system
> space, there are already quite a few little modules sprinkled about.
> If you review the allocations that already exist on process creation,
> you'll see that there is quite a bit thats already allocated. (again this
> is va space thats allocated.) A simple "lm" after loading a small native
> app in ntsd (say "ntsd calc.exe") shows:
> 0:000> lm
> start end module name
> 01000000 0101f000 calc (deferred)
> 77290000 772d9000 SHLWAPI (deferred)
> 77380000 77b5d000 SHELL32 (deferred)
> 77ba0000 77bf4000 msvcrt (deferred)
> 77c00000 77c44000 GDI32 (deferred)
> 77c50000 77cf4000 RPCRT4 (deferred)
> 77d00000 77d8f000 USER32 (deferred)
> 77da0000 77e30000 ADVAPI32 (deferred)
> 77e40000 77f34000 kernel32 (deferred)
> 77f40000 77ffa000 ntdll (pdb symbols)
>
> (yes.. thats a 2gb machine... i dont have my 3gb machine handy right now)
> Meaning that the largest linear region of memory inside 2gb we could
> allocate at this point would be 0x76271000 or 1.9 gigs.
> When you have a more complicated app in memory, (say something running
> managed code) you have many more dll's loaded at various places in memory.
>
> That said, the gc is only able to allocate from the pool of free memory,
> and have a few hundred megs of virtual address space consumed when an app
> is running is not uncommong. (thats va space. so, it just means that its
> atleast reserved, and maybe not actually allocated). On some Oses
(depends
> on what all is being loaded into a process) your ability to allocate
> certain amounts of memory will vary. So, for instance running inside a
> debugger will lower the amount you'll be able to allocate.
>
> I'd expect that in general you'll have better luck trying to allocate
> smaller chunks. But, dont expect to be able to get the full 3gb (with
/3gb
> enabled) or the full 2gb with normal 2gb enabled.
>
> -scott
>
> --------------------
> >From: "Alec Seward" <nospam@please.org>
> >Subject: Re: .NET Framework Can't use Available Memory
> >Date: Thu, 17 Jul 2003 10:57:01 +0200
> >
> >Jim,
> >
> >I have managed to squeeze out 2.4 Gb to one process in NET v1.1, on a
4Gb
> >computer running WinXP Pro.
> >
> >1. Add the /3GB switch in boot.ini (as you did)
> >2. After building you program run the linker (located in e.g. C:\Program
> >Files\Microsoft Visual Studio .NET 2003\Vc7\bin)
> >link -edit -LARGEADDRESSAWARE target.exe
> >where "target.exe" is your application executable
> >
> >Now run the executable and you should be able to allocate up to 2.4Gb to
> the
> >process.
> >Why it stops there and don't reach 3Gb, I don't know.
> >An unmanaged process can of course access all of the available 3Gb of
user
> >memory in Windows).
> >
> >Does anyone know if it is possible to do this in .NET/CLR as well?
> >
> >
> >Best
> >
> >Alec Seward
> >
> >"Memory limitations, memory limitations...why are there so many of them?"
> >
> >----- Original Message -----
> >From: "Jim Snape" <jsjunk_2002.deletethiss@yahoo.co.uk>
> >Newsgroups: microsoft.public.dotnet.framework
> >Sent: July 15, 2003 14:31
> >Subject: .NET Framework Can't use Available Memory
> >
> >
> >> I need to cache large amounts of data for fast lookup, hence I've been
> >> running some tests on various operating systems (XP, Win2k Adv etc) and
> it
> >> seems that no matter how much physical memory is available a .NET
> >framework
> >> application can not use more that 1.2GB. I have even tried with a 4GB
> >> machine using the 3GB boot time switch.
> >>
> >> Is there some hard limit built into the framework? I can't believe it
> >would
> >> be handicapped in this manor. Is there any way of circumventing the
> limit?
> >>
> >> Thanks in advance,
> >> Jim
> >>
> >> p.s. For reference, the code I used to test the limit is:
> >>
> >> using System;
> >>
> >> namespace MemAlloc
> >> {
> >> class MemBlock
> >> {
> >> private MemBlock next;
> >> private byte[] data = new byte[1024*64];
> >>
> >> const int OneMeg = 1024 * 1024;
> >>
> >> static void Main(string[] args)
> >> {
> >> try
> >> {
> >> MemBlock first = new MemBlock();
> >>
> >> while (true)
> >> {
> >> MemBlock next = new MemBlock();
> >> next.next = first;
> >> first = next;
> >>
> >> Console.WriteLine("MEM: {0} MB",
> >> GC.GetTotalMemory(false)/OneMeg);
> >> }
> >> }
> >> catch (Exception ex)
> >> {
> >> Console.WriteLine("{0}: {1}", ex.Source, ex.Message);
> >> }
> >> }
> >> }
> >> }
> >>
> >>
> >
> >
> >
>
>
> --
>
> This posting is provided "AS IS" with no warranties, and confers no
rights.
> Use of included script samples are subject to the terms specified at
> http://www.microsoft.com/info/cpyright.htm
>
> Note: For the benefit of the community-at-large, all responses to this
> message are best directed to the newsgroup/thread from which they
> originated.
>

Scott,

here are the last output lines of the C and the C# programs before memory
allocation failure and program crash (C: malloc failed, C#:
OutOfMemoryException)
All apps were compiled using VStudio 2003 EA.

C used the /LARGEADDRESSAWARE linker option.
In the C# case the app was relinked
link -edit -LARGEADDRESSAWARE Memtest.exe (why isn't there a linker options
in VStudio for C# projects?)

As you can see, C# version falls short of 600Mb (with LARGEADDRESSAWARE).
I should add that I've tried several other code snippets with the same
results.

What do you think... what is causing the difference??

Program output before crash and the both code snippets attached below.....
Scott, could you please also describe your test setup?

/Alec Seward

---- Native C, no /LARGEADDRESSAWARE ----
1996000 Kb allocated
process size according to taskManager: 2.0Gb

---- Native C, with /LARGEADDRESSAWARE ----
2998000 Kb allocated
process size according to taskManager: : 3.0Gb

---- C#, no /LARGEADDRESSAWARE ----
1455000 Kb allocated
process size according to taskManager: : 1.5Gb

---- C#, with /LARGEADDRESSAWARE ----
2328000 Kb allocated
process size according to taskManager: : 2.4Gb


---------- Native C version -----------------------
#include <stdio.h>
#include <stdlib.h>
void TestMemoryLimit() {
int *x;
for (int i = 0; i < 10000000; i ++) {
x = (int*) malloc(256 * sizeof(int)); //allocate 1K array
if (x == NULL) {
printf("Malloc failed\n");
break;
}
if (i % 1000 == 0) {
printf("%d Kb allocated\n", i);
}
}
}

void main() {
printf("Hello\n");
TestMemoryLimit();
}

---------- Managed C# version -----------------------
using System;
public class MemTest {

static void TestMemoryLimit() {
int[][] x = new int[10000000][];
for (int i = 0; i < x.Length; i ++) {
x[i] = new int[256]; //allocate 1K array
if (i % 1000 == 0) {
Console.WriteLine(i + " Kb allocated");
}
}
}
public static void Main() {
TestMemoryLimit();
}

}

---------- End -----------------------

"Scott Wadsworth [MSFT]" <bwadswor@online.microsoft.com> wrote in message
news:LM1aiIZTDHA.1924@cpmsftngxa06.phx.gbl...
> 600megs is quite a big difference. I dont see anything that bad on my
> machines.
> Can you drop me the exact apps you are comparing?
>
> in using a small variant of whats shown below, i see a managed version
that
> eats 1mb at a time, peak out at 1755mb.
> A native version of bascially the same thing is able to allocate about
> 1822mb of native 1mb chunks.
>
> If I descrease the block size down to 64k i can allocate ~1820mb in the
> managed process.
> And the same attempt in native yields a peak of about 1900mb.
>
> On one of my 3gb boxes the most i was able to allocate was 2.4gb managed,
> and 2.5gb native. (in 1mb chunks).
>
> The peak that i can allocate native(and managed) varies quite a bit from
> machine to machine.
>
> Its important to remeber a few things when comparing managed with native.
> 1. GC.GetTotalMemory is reporting the number of user visable bytes in
> the gc allocated memory. (each will have a small overhead. so if you
have
> a million objects, expect a few megs of overhead. etc.)
> 2. The space "consumed" in memory by dll's is only the VA space. What
> this means is that address space(a chunk of memory) can really be in the
> three fundamental states.
> A. Allocated and commited - Meaning that you have done a new and
> that the memory can be directly written. It might consume physical ram,
or
> it might consume space on disk in a swap file.
> B. Allocated and not commited - When a dll is mapped into memory for
> instance, address space is reserved for it (depending on the situation,
the
> dll might need a big linear region.), but since we didnt load the whole
> dll, we never need to commit the space.
> C. Free - this memory is open to be allocated. Though it might be
> fragmented, and we might only have little 64k chunks all over the place
> that are free.
>
> The GC is likely not the reason for the 600meg diff you are seeing. Drop
> me the little snippets that make up the example, and i'd love to take a
> detailed look and let you know whats going on.
>
> thanks!
> -Scott
>
>
> --------------------
> >From: "Alec Seward" <nospam@please.org>
> >Subject: Re: .NET Framework Can't use Available Memory
> >Date: Sat, 19 Jul 2003 00:03:44 +0200
> >
> >Scott, thanks for your answer.
> >
> >I cannot argue with your reasoning, but does it really give the whole
> >picture??
> >
> >When running a C program (WinXp with 4Gb 3GB swittch), allocation (in
small
> >chunks) can continue until the whole process consumes 3Gb (quite on the
> byte
> >actually).
> >When switching to C#, doing the same thing, it is impossible (at least
for
> >me) to get past the 2.4Gb barrier for the whole process. Note that this
is
> >the total memory for the process (as in the C case), not just what's been
> >allocated.
> >
> >600 MB is missing here!! Running a minimal console app in C# lowers the
> >ceiling of available RAM compared to unmanaged code by 600 MBytes . And
you
> >are saying that all this wasted memory is is eaten up by loaded NET DLL's
> >and the GC ??!? Many computers does not even have that much RAM.
> >
> >Please tell me there is a workaround. Does really a minimalistic ten line
> C#
> >program (that only allocates integers) really require 600 Mb extra of
> loaded
> >DLL's, or....is it instead so "complicated" that the GC must reserve all
> >those bytes just in case.
> >
> >If the GC is the devil here, I rather see it reserve less memory (and
take
> >the risk of not be able to garbage collect my simple int allocations),
> >instead of my app crashing with an OutOfMemoryException with 600 Mb still
> >unused.
> >
> >I missing something here?
> >
> >
> >Regards
> >/Alec Seward
> >
> >
> >
> >
> >"Scott Wadsworth [MSFT]" <bwadswor@online.microsoft.com> wrote in message
> >news:lSBjK9WTDHA.1924@cpmsftngxa06.phx.gbl...
> >> This behavior you are seeing is normal by design.
> >>
> >> The /3gb switch and LARGEADDRESSAWARE link flag cause the OS to allow a
> >> process to use 3gb of addressable user space.
> >>
> >> (history for any other readers of this thread)
> >> Each "normal" process has 4gb of address space (consider a 32-bit
> >pointer).
> >> There are extentions that allow you slide a window through more
memory,
> >> but fundamenatlly you cannot escape the fact that common x86 machines
> only
> >> support 32-bit addresses (therefore 4gb). Normally the OS uses the top
> >> half of memory (virtual address space) for system space. (i.e. thats
> >where
> >> the the "kernel" lives). If you pass /3GB in the boot.ini as a flag to
> >the
> >> kernel, it will an extra check during the loading of a process. This
> >extra
> >> check will look for the LARGEADDRESSAWARE flag (settable by linker
option
> >> or with imagecfg). If that flag is set, the OS will erect a process
with
> >> 3gb user space, and 1gb system space. However, through that 3gb of
> system
> >> space, there are already quite a few little modules sprinkled about.
> >> If you review the allocations that already exist on process creation,
> >> you'll see that there is quite a bit thats already allocated. (again
> this
> >> is va space thats allocated.) A simple "lm" after loading a small
native
> >> app in ntsd (say "ntsd calc.exe") shows:
> >> 0:000> lm
> >> start end module name
> >> 01000000 0101f000 calc (deferred)
> >> 77290000 772d9000 SHLWAPI (deferred)
> >> 77380000 77b5d000 SHELL32 (deferred)
> >> 77ba0000 77bf4000 msvcrt (deferred)
> >> 77c00000 77c44000 GDI32 (deferred)
> >> 77c50000 77cf4000 RPCRT4 (deferred)
> >> 77d00000 77d8f000 USER32 (deferred)
> >> 77da0000 77e30000 ADVAPI32 (deferred)
> >> 77e40000 77f34000 kernel32 (deferred)
> >> 77f40000 77ffa000 ntdll (pdb symbols)
> >>
> >> (yes.. thats a 2gb machine... i dont have my 3gb machine handy right
> now)
> >> Meaning that the largest linear region of memory inside 2gb we could
> >> allocate at this point would be 0x76271000 or 1.9 gigs.
> >> When you have a more complicated app in memory, (say something running
> >> managed code) you have many more dll's loaded at various places in
> memory.
> >>
> >> That said, the gc is only able to allocate from the pool of free
memory,
> >> and have a few hundred megs of virtual address space consumed when an
app
> >> is running is not uncommong. (thats va space. so, it just means that
its
> >> atleast reserved, and maybe not actually allocated). On some Oses
> >(depends
> >> on what all is being loaded into a process) your ability to allocate
> >> certain amounts of memory will vary. So, for instance running inside a
> >> debugger will lower the amount you'll be able to allocate.
> >>
> >> I'd expect that in general you'll have better luck trying to allocate
> >> smaller chunks. But, dont expect to be able to get the full 3gb (with
> >/3gb
> >> enabled) or the full 2gb with normal 2gb enabled.
> >>
> >> -scott
> >>
> >> --------------------
> >> >From: "Alec Seward" <nospam@please.org>
> >> >Subject: Re: .NET Framework Can't use Available Memory
> >> >Date: Thu, 17 Jul 2003 10:57:01 +0200
> >> >
> >> >Jim,
> >> >
> >> >I have managed to squeeze out 2.4 Gb to one process in NET v1.1, on a
> >4Gb
> >> >computer running WinXP Pro.
> >> >
> >> >1. Add the /3GB switch in boot.ini (as you did)
> >> >2. After building you program run the linker (located in e.g.
C:\Program
> >> >Files\Microsoft Visual Studio .NET 2003\Vc7\bin)
> >> >link -edit -LARGEADDRESSAWARE target.exe
> >> >where "target.exe" is your application executable
> >> >
> >> >Now run the executable and you should be able to allocate up to 2.4Gb
to
> >> the
> >> >process.
> >> >Why it stops there and don't reach 3Gb, I don't know.
> >> >An unmanaged process can of course access all of the available 3Gb of
> >user
> >> >memory in Windows).
> >> >
> >> >Does anyone know if it is possible to do this in .NET/CLR as well?
> >> >
> >> >
> >> >Best
> >> >
> >> >Alec Seward
> >> >
> >> >"Memory limitations, memory limitations...why are there so many of
> them?"
> >> >
> >> >----- Original Message -----
> >> >From: "Jim Snape" <jsjunk_2002.deletethiss@yahoo.co.uk>
> >> >Newsgroups: microsoft.public.dotnet.framework
> >> >Sent: July 15, 2003 14:31
> >> >Subject: .NET Framework Can't use Available Memory
> >> >
> >> >
> >> >> I need to cache large amounts of data for fast lookup, hence I've
been
> >> >> running some tests on various operating systems (XP, Win2k Adv etc)
> and
> >> it
> >> >> seems that no matter how much physical memory is available a .NET
> >> >framework
> >> >> application can not use more that 1.2GB. I have even tried with a
4GB
> >> >> machine using the 3GB boot time switch.
> >> >>
> >> >> Is there some hard limit built into the framework? I can't believe
it
> >> >would
> >> >> be handicapped in this manor. Is there any way of circumventing the
> >> limit?
> >> >>
> >> >> Thanks in advance,
> >> >> Jim
> >> >>
> >> >> p.s. For reference, the code I used to test the limit is:
> >> >>
> >> >> using System;
> >> >>
> >> >> namespace MemAlloc
> >> >> {
> >> >> class MemBlock
> >> >> {
> >> >> private MemBlock next;
> >> >> private byte[] data = new byte[1024*64];
> >> >>
> >> >> const int OneMeg = 1024 * 1024;
> >> >>
> >> >> static void Main(string[] args)
> >> >> {
> >> >> try
> >> >> {
> >> >> MemBlock first = new MemBlock();
> >> >>
> >> >> while (true)
> >> >> {
> >> >> MemBlock next = new MemBlock();
> >> >> next.next = first;
> >> >> first = next;
> >> >>
> >> >> Console.WriteLine("MEM: {0} MB",
> >> >> GC.GetTotalMemory(false)/OneMeg);
> >> >> }
> >> >> }
> >> >> catch (Exception ex)
> >> >> {
> >> >> Console.WriteLine("{0}: {1}", ex.Source,
ex.Message);
> >> >> }
> >> >> }
> >> >> }
> >> >> }
> >> >>
> >> >>
> >> >
> >> >
> >> >
> >>
> >>
> >> --
> >>
> >> This posting is provided "AS IS" with no warranties, and confers no
> >rights.
> >> Use of included script samples are subject to the terms specified at
> >> http://www.microsoft.com/info/cpyright.htm
> >>
> >> Note: For the benefit of the community-at-large, all responses to this
> >> message are best directed to the newsgroup/thread from which they
> >> originated.
> >>
> >
> >
> >
>
>
> --
>
> This posting is provided "AS IS" with no warranties, and confers no
rights.
> Use of included script samples are subject to the terms specified at
> http://www.microsoft.com/info/cpyright.htm
>
> Note: For the benefit of the community-at-large, all responses to this
> message are best directed to the newsgroup/thread from which they
> originated.
>