Memory Limit for Visual Studio 2005???

[Peter Olcott]

It looks like System::Collections::Generic.List throws and OUT_OF_MEMORY
exception whenever memory allocated exceeds 256 MB. I have 1024 MB on my system
so I am not even out of physical RAM, much less virtual memory.

Are other people experiencing this same problem?

 

[Michael Nemtsev]

Hello Peter,

Memmory is allocated by blocks 32/64 mb, so if your GC is fragmented FW may
not found free space to allocate aditional block, albeit you can have enough
memmory

PO> It looks like System::Collections::Generic.List throws and
PO> OUT_OF_MEMORY exception whenever memory allocated exceeds 256 MB. I
PO> have 1024 MB on my system so I am not even out of physical RAM, much
PO> less virtual memory.
PO>
PO> Are other people experiencing this same problem?
PO>
---
WBR,
Michael Nemtsev [C# MVP] :: blog: http://spaces.live.com/laflour

"The greatest danger for most of us is not that our aim is too high and we
miss it, but that it is too low and we reach it" (c) Michelangelo

 

[John J. Hughes II]

I checked in task manager and my peek memory usage for VS2005 is 288,060K
but then I have 2Gigs on this system.

Regards,
John

 

[Ignacio Machin \( .NET/ C# MVP \)]

Hi,

What kind of project is this?

IIRC a web app has a limit refering to the amount of memory it can consume

 

[Peter Olcott]

It may be a Visual Studio Express limitation

 

[Peter Olcott]

Try and add one gig of Bytes to a List<Byte> and see if it doesn't abnormally
terminate.

 

[Peter Olcott]

Its a regular desktop windows project under Visual Studio 2005 Express.

 

[Willy Denoyette [MVP]]

It looks like System::Collections::Generic.List throws and OUT_OF_MEMORY exception
whenever memory allocated exceeds 256 MB. I have 1024 MB on my system so I am not even out
of physical RAM, much less virtual memory.

Are other people experiencing this same problem?

Why do you mention VS2005? This is thrown at program run-time, right? The run-time
limitation is 2GB for a CLR object type, that means that your List can only hold at most
2GB, however, due heap fragmentation of a 32 bit process, the maximum size of a single
object is limited to something from 1.2GB up to 1.8GB depending on the type of OS and
application. A simple console application should be able to allocate a single List of
~1.6GB or more , Windows Forms will top at ~1.2GB for the largest List (or array or
whatever).

Willy.

 

[Peter Olcott]

Why do you mention VS2005? This is thrown at program run-time, right? The
run-time limitation is 2GB for a CLR object type, that means that your List
can only hold at most 2GB, however, due heap fragmentation of a 32 bit
process, the maximum size of a single object is limited to something from
1.2GB up to 1.8GB depending on the type of OS and application. A simple
console application should be able to allocate a single List of ~1.6GB or more
, Windows Forms will top at ~1.2GB for the largest List (or array or
whatever).

Willy.

If you have actually tested this and thus know that it works empirically rather
than theoretically, then this must be a limitation of Visual Studio 2005
Express. I could not get Visual Studio 2005 Express to allocate more than 256 MB
without abnormally terminating.

I was very pleased with its relative performance to Native Code Visual C++ 6.0.
It was something like 50% faster on every test.

 

[Willy Denoyette [MVP]]

If you have actually tested this and thus know that it works empirically rather than
theoretically, then this must be a limitation of Visual Studio 2005 Express. I could not
get Visual Studio 2005 Express to allocate more than 256 MB without abnormally
terminating.

I was very pleased with its relative performance to Native Code Visual C++ 6.0. It was
something like 50% faster on every test.

No, once again this is run-time related VS does (and can't impose such restrictions) , try
to compile and run your code from the command-line prompt if you do't trust me.
Keep in mind that when you don't pre-allocate the List, you will end with a List doubling
it's size each time it get's filled to the max. capacity, so the OOM exception might get
thrown when no free *contiguous* block of ~512MB can be found in excess of the already
allocated ~256MB.

Willy.





Willy.

 

[Peter Olcott]

No, once again this is run-time related VS does (and can't impose such
restrictions) , try to compile and run your code from the command-line prompt
if you do't trust me.
Keep in mind that when you don't pre-allocate the List, you will end with a
List doubling it's size each time it get's filled to the max. capacity, so the
OOM exception might get thrown when no free *contiguous* block of ~512MB can
be found in excess of the already allocated ~256MB.

Willy.





Willy.

It works fine for native code std::vector, yet does not work for either managed
std::vector or List<Byte> . The command line version failed to compile.

 

[Chris Mullins]

Peter Olcott Wrote:

PO> It looks like System::Collections::Generic.List throws and
PO> OUT_OF_MEMORY exception whenever memory allocated exceeds 256 MB. I
PO> have 1024 MB on my system so I am not even out of physical RAM, much
PO> less virtual memory.

I was curious if there was a limt there, so I wrote this:

private void button1_Click(object sender, EventArgs e)
{
int bytesPerArray = 1024 * 32; // 32k per byte. No large object heap
interaction.
long totalBytes = 0;
List<byte[]> bytes = new List<byte[]>();
while (true)
{
byte[] b = new byte[bytesPerArray];
bytes.Add(b);
totalBytes += bytesPerArray;
System.Diagnostics.Debug.WriteLine("Total: " +
totalBytes.ToString());
}
}

I compiled this as an x86 application, and ran it.

In my output window, the last things in there was:
Total: 1704427520
A first chance exception of type 'System.OutOfMemoryException'
occurred in WindowsApplication2.exe

This is exactly what I expected to see, as I know that each 32-bit Windows
Process gets 4GB of virual memory space, and that 4GB is split in half,
giving 2GB to user code to play with. The managed heap can typically (if
it's not fragmented) get up to 1.5-1.7 gigabytes before issues arise.

If I compile this for x64 (or leave it as Any CPU), the limites are much
higher.

My machine does have 4GB of memory on it, but I've seen these exact same
results on machines with far less memory. (I build very big, very scalable,
applications all day long... and running into memory limitiations in 32-bit
land was a big problem I had for years).

 

[Chris Mullins]

Try and add one gig of Bytes to a List<Byte> and see if it doesn't
abnormally terminate.

Well, I did just that (got to 1.7GB) and it ran fine as an x86 application.
I was allocating 32KB byte chunks though.

This is a VERY different use case from allocating a single 1GB buffer. I
tried to allocate a single 1GB array:
private void button2_Click(object sender, EventArgs e)
{
List<byte[]> bb = new List<byte[]>();
int GB = 1024 * 1024 * 1024;
byte[] b = new byte[GB];
bb.Add(b);
MessageBox.Show("Allocated 1 GB");
}

.... and this immediatly threw an OOM when I compiled it under x86.
It did run perfectly (and instantly) when compiled and run under x64.

Not respective of platform, if you're really allocating memory in chunks
this big, you need to rethink your algorithm.

Even in x64 land, holding onto chunks in the heap this big would be scary.

 

[Chris Mullins]

If you have actually tested this and thus know that it works empirically
rather than theoretically, then this must be a limitation of Visual Studio
2005 Express. I could not get Visual Studio 2005 Express to allocate more
than 256 MB without abnormally terminating.

I have explicitly tested this, and it works fine.

I've been writing .Net applications that max out .Net Memory Management for
years now, and know it to work fine.

You keep missing the fact that VS2005 isn't a compiler. It's using the
standard CSC compiler that comes with the .Net framework. Even this isn't
really a compiler, as it only spits out IL.

The Jitter is what "compiles" that IL to native code and executes it. How
you generated that IL doesn't matter. The jitter also comes with the .Net
framework, and isn't a part of VS2005 Express.

I was very pleased with its relative performance to Native Code Visual C++
6.0. It was something like 50% faster on every test.

Lik everything else, it depends on your use cases. In my experience, the
performance different between native C++ and managed C# has always been
"close enough" to side with C# due to the increase in developer
productivity.

 

[Peter Olcott]

Try and add one gig of Bytes to a List<Byte> and see if it doesn't abnormally
terminate.

Well, I did just that (got to 1.7GB) and it ran fine as an x86 application. I
was allocating 32KB byte chunks though.

This is a VERY different use case from allocating a single 1GB buffer. I tried
to allocate a single 1GB array:
private void button2_Click(object sender, EventArgs e)
{
List<byte[]> bb = new List<byte[]>();
int GB = 1024 * 1024 * 1024;
byte[] b = new byte[GB];
bb.Add(b);
MessageBox.Show("Allocated 1 GB");
}

... and this immediatly threw an OOM when I compiled it under x86.
It did run perfectly (and instantly) when compiled and run under x64.

Not respective of platform, if you're really allocating memory in chunks this
big, you need to rethink your algorithm.

Would you think that 18,000 hours would be enough thought? That's how much I
have into it.

 

[Peter Olcott]

If you have actually tested this and thus know that it works empirically
rather than theoretically, then this must be a limitation of Visual Studio
2005 Express. I could not get Visual Studio 2005 Express to allocate more
than 256 MB without abnormally terminating.

I have explicitly tested this, and it works fine.

I've been writing .Net applications that max out .Net Memory Management for
years now, and know it to work fine.

You keep missing the fact that VS2005 isn't a compiler. It's using the
standard CSC compiler that comes with the .Net framework. Even this isn't
really a compiler, as it only spits out IL.

I have written a couple of compilers, and the jitter is not a compiler. The
tricky part is translating the nested do-while and if-then-else statements
comprised of compounds relational expressions into jump code. The jitter does
not need to do this, this part is already done. All the jitter has to do is to
translate pseudo assembly language into machine code.

 

[Willy Denoyette [MVP]]

It works fine for native code std::vector, yet does not work for either managed
std::vector or List<Byte> . The command line version failed to compile.

There is no such thing like MANAGED std::vector, and the program should compile just fine
from the command line. if it compiles from VS. Both VS and the csc.exe are both driving the
same compiler.

Willy.

 

[Peter Olcott]

Try this simpler case:
uint SIZE = 0x3FFFFFFF; // 1024 MB

List<uint> Temp;
for (uint N = 0; N < SIZE; N++)
Temp.Add(N);

Mine now bombs out on just short of half my memory. I am guessing that it runs
out of actual RAM when it doubles the size on the next reallocation. Unlike the
native code compiler, the managed code compiler must have actual RAM, virtual
memory will not work.

 

[Peter Olcott]

There is no such thing like MANAGED std::vector,

Yes, there is just recently. This saves old C++ programmers like me alot of
learning curve switching to .NET.

 

[Willy Denoyette [MVP]]

Try and add one gig of Bytes to a List<Byte> and see if it doesn't abnormally terminate.

Well, I did just that (got to 1.7GB) and it ran fine as an x86 application. I was
allocating 32KB byte chunks though.

This is a VERY different use case from allocating a single 1GB buffer. I tried to allocate
a single 1GB array:
private void button2_Click(object sender, EventArgs e)
{
List<byte[]> bb = new List<byte[]>();
int GB = 1024 * 1024 * 1024;
byte[] b = new byte[GB];
bb.Add(b);
MessageBox.Show("Allocated 1 GB");
}

... and this immediatly threw an OOM when I compiled it under x86.
It did run perfectly (and instantly) when compiled and run under x64.

Not respective of platform, if you're really allocating memory in chunks this big, you
need to rethink your algorithm.

Even in x64 land, holding onto chunks in the heap this big would be scary.

Windows forms needs some native dll's that are loaded by the OS loader at an address which
further fragments the process heap, and because the GC heap allocates from the process heap,
it cannot allocate a contiguous area larger than the largest process heap fragment. The net
result is that the most simple Windows programs (native and managed) cannot allocate more
than ~1.2 GB or less , depending on OS version SP and OS language version.

Willy.