Huge arrays?

[Michael Gray]

VS 2003
VB.net
Win2000 SP4

The System.Array class seems to be limited to 32 bit addresses,
meaning that one can only assign 2^32 elements.

Is there any way that I can have an array that allows 2^64 elements?
Or doesn't the CLR support this on a 32 bit opsys?
I suspect that I am going to have to make such a beastie myself...

What about sparse arrays?

 

[Robin Tucker]

Can I ask what you want to use such a beast for?

 

[Inge Henriksen]

Use a database to store this many elements, as collections are also limited
by an Integer. You could ofcourse use several arrays or collections.

 

[Jay B. Harlow [MVP - Outlook]]

Michael,
In addition to the other comments:

| The System.Array class seems to be limited to 32 bit addresses,
| meaning that one can only assign 2^32 elements.
..NET 1.1 has "support" for large arrays in that there is an Array.LongLength
property (Int64) & Array.GetValue & Array.SetValue are overloaded for Long
(Int64).

http://msdn.microsoft.com/library/d...html/frlrfSystemArrayClassLongLengthTopic.asp

http://msdn.microsoft.com/library/d.../html/frlrfSystemArrayClassGetValueTopic3.asp

| Is there any way that I can have an array that allows 2^64 elements?
| Or doesn't the CLR support this on a 32 bit opsys?
However! other then sparse arrays, which I'm not sure how much support
there is, how would you actually allocate said array? Remember that under 32
bit OS you can only index 4G of memory, of which the OS wants to keep 2G (1G
on some OSes) for itself, the runtime, your code, the stack, and other data
is going to use up part of the usable 2G, so the actually size of an Array
is going to be much smaller.

If I truly needed an "array" that is larger the 512M or so, I would consider
defining a new Type that used a file for the actual backing of the data &
read & write each element as needed. I would consider caching the elements
if performance became a problem. You could define this Type such that it
behaved like most normal collections.

Alternatively I have defined "Sparse Arrays", where I keep the "index" &
elements in a HashTable...


BTW: .NET 2.0 http://lab.msdn.microsoft.com/vs2005/ includes both a 32bit &
64bit runtime, so on a 64bit OS with the 64bit runtime you can have
obnoxiously large arrays... Whether you should or not is another discussion
;-)

Hope this helps
Jay

| VS 2003
| VB.net
| Win2000 SP4
|
| The System.Array class seems to be limited to 32 bit addresses,
| meaning that one can only assign 2^32 elements.
|
| Is there any way that I can have an array that allows 2^64 elements?
| Or doesn't the CLR support this on a 32 bit opsys?
| I suspect that I am going to have to make such a beastie myself...
|
| What about sparse arrays?

 

[Michael Gray]

Michael,
In addition to the other comments:

| The System.Array class seems to be limited to 32 bit addresses,
| meaning that one can only assign 2^32 elements.
.NET 1.1 has "support" for large arrays in that there is an Array.LongLength
property (Int64) & Array.GetValue & Array.SetValue are overloaded for Long
(Int64).

http://msdn.microsoft.com/library/d...html/frlrfSystemArrayClassLongLengthTopic.asp

http://msdn.microsoft.com/library/d.../html/frlrfSystemArrayClassGetValueTopic3.asp

| Is there any way that I can have an array that allows 2^64 elements?
| Or doesn't the CLR support this on a 32 bit opsys?
However! other then sparse arrays, which I'm not sure how much support
there is, how would you actually allocate said array? Remember that under 32
bit OS you can only index 4G of memory, of which the OS wants to keep 2G (1G
on some OSes) for itself, the runtime, your code, the stack, and other data
is going to use up part of the usable 2G, so the actually size of an Array
is going to be much smaller.

If I truly needed an "array" that is larger the 512M or so, I would consider
defining a new Type that used a file for the actual backing of the data &
read & write each element as needed. I would consider caching the elements
if performance became a problem. You could define this Type such that it
behaved like most normal collections.

Alternatively I have defined "Sparse Arrays", where I keep the "index" &
elements in a HashTable...


BTW: .NET 2.0 http://lab.msdn.microsoft.com/vs2005/ includes both a 32bit &
64bit runtime, so on a 64bit OS with the 64bit runtime you can have
obnoxiously large arrays... Whether you should or not is another discussion
;-)

Hope this helps
Jay

Ah, that's the answer that I was looking for...
Thanks to the other posters as well.

I knew that this rather strange request would prompt the same reaction
that I initially had: Why? and wouldn't you be better off doing it
another way?
The answers to which are: It was an enquiry to eliminate lingering
doubts, and Yes.

I really only wanted huge sparse arrays, ala FORTRAN, but noticed
these 64 bit addressing functions for System.Array objects(Getvalue &
SetValue), but was then thrown by not being able to index them as
such, except via these special functions.

They "appeared" to have 64 bit addressing, only very much "under the
hood" as it were.

It just seemed strange to me that Microsoft would go to all the
trouble of allowing 64 bit addresses in System.Array, and then going
through hoops to cover it up, by forcing the default indexing schemes
to 32 bit! (At least in VB.Net anyway)

Thanks for all your replies, problem will be solved another way...

 

[Michael Gray]

Can I ask what you want to use such a beast for?

I don't really, but want huge sparse arrays, such as FORTRAN provides.
See my other reply for more info...

 

[Michael Gray]

Use a database to store this many elements, as collections are also limited
by an Integer. You could ofcourse use several arrays or collections.

:
Thanks for taking the time to reply.
See my lengthier response to J.B. Harlow for more details, if you are
interested.