Assembly memory limit exception -WHY?

[Guest]

Hi,

I wrote an .NET application that allocate a large byte array.
If the array is of size 700 MByte, it's ok and no error is thrown.
But when I try to allocate a 800 MByte byte array, an OutOfMemory Exception
is thrown.

The PC that this application is running on has a 2 Giga bytes RAM and the
free memory is 1.3 Giga bytes when the application was trying to allocate the
byte array.
The PC has two disks, both are set to use virtual memory (large one).

It's looks like the system is limiting each assembly to use that much memory.
Is there a way to change this settings to allow a larger amount of memory
for each assembly.
If so, How do I do that?

 

[Nicholas Paldino [.NET/C# MVP]]

Sharon,

I don't think that is necessarily the problem. With arrays, the memory
has to be continuous. Asking for 700-800 MB of continuous memory is a lot,
and to be quite frank, not the best way to go about this.

I would look into creating a container solution which would break this
down into smaller groups which you could access (using arrays of a smaller
size internally, which would do some math to determine which array is being
accessed by the index passed in).

AFAIK, there isn't a setting per assembly indicating how much memory
that instances from types in that assembly can use.

Hope this helps.

 

[Guest]

I don't this this is the case.

I found anoter discussion about the same probelm is here:

http://msdn.microsoft.com/newsgroup...harp&mid=d05f7a58-f4f2-4bb4-a43f-0c3b9681dd2b

 

[Guest]

I don't think that is necessarily the problem. With arrays, the memory
has to be continuous. Asking for 700-800 MB of continuous memory is a lot,
and to be quite frank, not the best way to go about this.

I'm not sure that's it. IIRC the memory is only guaranteed contigious
within the applications address space, and that there's no way to assure that
it maps to contigious blocks at the OS level.

 

[Nicholas Paldino [.NET/C# MVP]]

Sharon,

Based on the thread that you posted, yes, that is exactly the case.
Each of the responses indicates that the problem is assigning an array,
which is a contiguous space in memory. The solution posted there ^might^
work, only because it raises the amount of space that the process can use
(the 3GB option). However, the problem is still the same, and I would
recommend breaking it into smaller pieces and accessing those pieces.

 

[Willy Denoyette [MVP]]

I'm not sure that's it. IIRC the memory is only guaranteed contigious
within the applications address space, and that there's no way to assure
that
it maps to contigious blocks at the OS level.

Sure it is, and that is exactly what Nicolas meant, the address space
allocated to a process (VAS) is limited to 2GB on 32 bit windows (or 3GB
when 3GB tuning is enabled).

This 2GB address space is used to load all application components:
1. C Runtime DLL's, CLR DLL's and Windows system DLL's and private user
DLL's (if any)
2. The native heap space allocated by the system DLL's and the C runtime and
the CLR.
3. The managed heap allocated by the CLR
4. The loaded managed assemblies (note these aren't loaded as normal win32
DLL's).
5. The compiled pages from the JIT compilation of 4.

The results is that a process that hosts the CLR has ~1.5 MB of total free
heap space at process start, however this space is guaranteed to be
fragmented and the largest fragment is not going to be larger than ~1.2 GB.
It might be a lot smaller if some DLL gets mapped in the middle of this
address space.
From this 1.2GB the CLR will allocate it's Large Object heap, which by
itself might get fragmented because this LOH is never compacted by the GC.
Now what the OP didn't say is how many objects did get allocated before he
creates that large 800MB object (an array?), also I'm not sure where he gets
this size from (a wild guess?).
Not that the popular ArrayList are really heap hungry, suppose the OP stores
its data in a Arraylist that gets filled up to it's full 700MB, when the
application tries to insert more data into the ArrayList , the list will
grow by it's standard factor 2, so the CLR will need 1.4GB to accommodate
the list, and will throw an OOM exception because this space isn't available
(as one large chunk).

Willy.