Large arrays in c#

[Clive Tooth]

This works ok...
const int n = 500000000;
int[] a = new int[n];
a[1] = 2;

This fails with "Exception of type 'System.OutOfMemoryException' was
thrown."...
const int n = 600000000;
int[] a = new int[n];
a[1] = 2;

There seems to be a limit of 2GB on any array.

# If there is such a limit, where is it stated?
# How do I declare an array of more than 536,870,912 (=2^29) integers?

My computer has 16GB of RAM.

I am using Visual C# 2010 Express, Windows 7 Home Premium Version 6.1
(build 7601: Service Pack 1)

Microsoft Visual Studio 2010
Version 10.0.30319.1 RTMRel
Microsoft .NET Framework
Version 4.0.30319 RTMRel

 

[Marcel Müller]

This works ok...
const int n = 500000000;
int[] a = new int[n];
a[1] = 2;

This fails with "Exception of type 'System.OutOfMemoryException' was
thrown."...
const int n = 600000000;
int[] a = new int[n];
a[1] = 2;

There seems to be a limit of 2GB on any array.

No .NET object can exceed 2GB size. The allocator does not support this.
Because of some internal overhead, the practical maximum is somewhat less.

# If there is such a limit, where is it stated?
# How do I declare an array of more than 536,870,912 (=2^29) integers?

I would not recommend to allocate that large memory in one block for
several reasons.
Dealing with fragmentation in objects of that dimension will add no
significant overhead. But dealing with address space fragmentation will
be a serious point. There are BigArray implementations available.

My computer has 16GB of RAM.

This does not count, since the allocation is done in virtual memory. As
long as you do not access all these elements, most operating systems
will not associate physical memory with the address space. So the
important point is that you have a 64 bit platform and a 64 bit runtime.


Marcel

 

[Matt]

[...]

# If there is such a limit, where is it stated?
# How do I declare an array of more than 536,870,912 (=2^29) integers?

I would not recommend to allocate that large memory in one block for
several reasons.
Dealing with fragmentation in objects of that dimension will add no
significant overhead. But dealing with address space fragmentation will
be a serious point. There are BigArray implementations available.

Address space fragmentation is not an issue on 64-bit Windows.  Not fora
"small" array as in this example, anyway.

If the OP is trying to allocate such a large object on 32-bit Windows, then
yes...that's a bad idea.  It's not even possible, due to the 2GB
per-process limit on virtual address space, but of course even large
allocations that are possible would be a bad idea.

But presumably this is for a 64-bit process running on 64-bit Windows.  A
2GB allocation is really no big deal in that environment.  There's a
ginormous amount of virtual address space to work with in that case (to use
the technical term ).

This does not count, since the allocation is done in virtual memory. As
long as you do not access all these elements, most operating systems
will not associate physical memory with the address space. So the
important point is that you have a 64 bit platform and a 64 bit runtime..

Yes, and it _ought_ to work.  The .NET array types even support 64-bit
array lengths (LongLength property) in theory.  The problem is that you
can't get .NET itself to allocate an array that large.

That's correct, it was a conscious decision on the part of the
designers.
You can read one of the guy's motivations here:

http://blogs.msdn.com/b/joshwil/archive/2005/08/10/450202.aspx

It also shows a pretty nice way to get around the problem

It's one of my pet peeves: 64-bit programs in .NET aren't really fully
64-bits.  You get the advantages of the larger address space and higher
per-process memory limits, but individual objects are still limited in size
(as discussed here).  It's very annoying.  

I agree, and likely it will be fixed in a future release. However,
realistically,
how often do you really want to be able to allocate 2G of memory in a
single
block? (And, as I type this, I can hear Bill Gates chuckling about his
256k
comment).

matt

 

[Marcel Müller]

[...]
That's correct, it was a conscious decision on the part of the
designers.

Well, I don't think anyone was suggesting that .NET left large object
allocations out accidentally!

Most likely it is an optimization.

This, I doubt. The article you referenced hedged a bit, but note that it
was written prior to the actual release of v2.0. 4.5 is about to come out,
and we still don't see any change in this particular issue.

It is not a question of the .NET version. It is only a question of the
VM. E.g. the Mono .NET VM does not have this restriction on 64 bit hosts.


Marcel

 

[Arne Vajhøj]

Yes, and it _ought_ to work. The .NET array types even support 64-bit
array lengths (LongLength property) in theory. The problem is that you
can't get .NET itself to allocate an array that large.

It's one of my pet peeves: 64-bit programs in .NET aren't really fully
64-bits. You get the advantages of the larger address space and higher
per-process memory limits, but individual objects are still limited in size
(as discussed here). It's very annoying.

It does not matter much that LongLength exist. If it did not
it could be added.

The question is what to do with all the code that uses
Length.

And because arrays and collections interact in many ways
then making arrays "64 bit capable" would also require
collections to become "64 bit capable".

And then there is serialization between 32 and 64 bit systems.

It will not be a simple change.

Arne

 

[Arne Vajhøj]

It is not a question of the .NET version. It is only a question of the
VM. E.g. the Mono .NET VM does not have this restriction on 64 bit hosts.

AFAIK then Mono does not officially support Windows in 64 bit mode.

Arne

 

[Clive Tooth]

Thanks very much to all contributors to this thread. I have now
implemented my own 'BigArray' class that allows me to create the large
structures that I want to use.

 

[Arne Vajhøj]

It "matters" inasmuch as it suggests someone was actually thinking about
array lengths> 2^32-1 elements, but did not go "all the way". Which was
the point of what I wrote.

Solving 1/10000 of the problem is not really solving anything.

Not only did they no go all the way - they made a step of
0.01 mm forward.

Such code is still limited to 2^32-1 elements. But first, note that the
limit is 2GB objects. You can have (many) fewer than 2^32-1 elements and
still take advantage of an object size larger than 2GB.

So you are not really talking about the same thing that the OP is asking
about.

No.

But given that I did not quote anything from original post, then
that should not surprise anyone.

Second, code that was limited to using the Length property would simply be
restricted to arrays with fewer than 2^32-1 elements. How best to enforce
this is debatable, but IMHO the best way would be to throw an exception if
the Length property is used for an array where the length is larger than an
int value can store.

I completely agree that exception would be necessary.

Most code would "just work". After all, it's unlikely (not impossible, but
not likely) an array would wind up with too many elements for the Length
property to be valid if the code wasn't already aware of larger arrays.

No. Very likely.

All libraries that take arrays or collections as input.

No one said it would. But it's well within the "normal" in terms of the
kind of development resources that are invested in .NET. Lots of new
features in .NET and C# have appeared that require much more work than
enabling object allocation sizes larger than 2GB and array/collection sizes
greater than 2^32-1 (noting that the two are related, but independent
issues).

MS could add this stuff in relative few hours.

But the effort by MS is just a tiny fraction of the problem.

Arne

 

[Jeff Johnson]

"Matt" wrote in message

(And, as I type this, I can hear Bill Gates chuckling about his 256k
comment).

A) It was 640K.

B) There's no real proof that Gates actually said this, but it lives on in
apocrypha....

 

[Radu Cosoveanu]

As you know, on 32 bits, the max addressing limit is 2 GB.
There are 2 solutions:
- using memory as bulk and addressing how much you want, if you really want to go on this direction, which unfortunatelly is not the good one.
- change solution using a storage system or a database.
Why do you need so much memory ?

Kind regards,
Radu Cosoveanu.