Maximum Size of Byte Array

  • Thread starter Thread starter Gerrit
  • Start date Start date
G

Gerrit

Hi all,

I'm getting an OutOfMemoryException when I initialize a byte array in
C# like this:

Byte[] test = new Byte[420000000];

I'm using ASP.NET 2.0. In ASP.Net 1.1 it works fine. So what am I doing
wrong?

What is the maximum length of a byte array?

Can somebody help me? Thanks in advance!

--
Gerrit Horeis

Software Developer
CI-Gate Development & Consulting GmbH
http://www.ci-gate.de
http://www.xira.de
http://www.bitbauer.de
 
Gerrit said:
I'm getting an OutOfMemoryException when I initialize a byte array in
C# like this:

Byte[] test = new Byte[420000000];

I'm using ASP.NET 2.0. In ASP.Net 1.1 it works fine. So what am I doing
wrong?

What is the maximum length of a byte array?

Can somebody help me? Thanks in advance!

My guess is that ASP.NET 2.0 has some extra constraints in terms of how
much memory your app is allowed to use, to prevent wayward apps from
killing the machine. If I may say so, that's a fairly extreme use of
memory...

I can't say I know much about ASP.NET configuration - perhaps someone
in the ASP.NET group could help you there - but my gut feeling is
that's where the problem is.
 
Thank you very much!

The strange on this behavior is that it sometimes occurs and sometimes
not.
But I'll try my luck in a ASP.NET group.

Thank you so far!

--
Gerrit Horeis

Software Developer
CI-Gate Development & Consulting GmbH
http://www.ci-gate.de
http://www.xira.de
http://www.bitbauer.de
Gerrit said:
I'm getting an OutOfMemoryException when I initialize a byte array in
C# like this:

Byte[] test = new Byte[420000000];

I'm using ASP.NET 2.0. In ASP.Net 1.1 it works fine. So what am I doing
wrong?

What is the maximum length of a byte array?

Can somebody help me? Thanks in advance!

My guess is that ASP.NET 2.0 has some extra constraints in terms of how
much memory your app is allowed to use, to prevent wayward apps from
killing the machine. If I may say so, that's a fairly extreme use of
memory...

I can't say I know much about ASP.NET configuration - perhaps someone
in the ASP.NET group could help you there - but my gut feeling is
that's where the problem is.
 
| Hi all,
|
| I'm getting an OutOfMemoryException when I initialize a byte array in
| C# like this:
|
| Byte[] test = new Byte[420000000];
|
| I'm using ASP.NET 2.0. In ASP.Net 1.1 it works fine. So what am I doing
| wrong?
|
| What is the maximum length of a byte array?
|
| Can somebody help me? Thanks in advance!
|
| --
| Gerrit Horeis
|
| Software Developer
| CI-Gate Development & Consulting GmbH
| http://www.ci-gate.de
| http://www.xira.de
| http://www.bitbauer.de
|

You should never assume that any portion of memory is available, even a
simple object allocation may fail. Now, as for your array of bytes,
theoretically the maximum size is ~2 billion bytes, but you will never be
able to allocate that size from a 32 bit process. Same goes for your 420
Mbytes array, allocating this from a 32 bit process will certainly lead to
OOM failures, as a result of heap fragmentation.
There is no guarantee that such an amount of contiguous memory space will be
available, certainly not when allocated in long running applications like
services (asp.net). For asp.net there is also a configuration parameter
which limits the total size of the process memory to a certain percentages
of the available virtual memory, while this is something to watch for, the
only solution to your problem, redesign your application in order to reduce
your memory consumption (this is a memory hog which doesn't scale at all),
or move to a 64 bit OS.

Willy.
 
Hi,


Gerrit said:
Hi all,

I'm getting an OutOfMemoryException when I initialize a byte array in
C# like this:

Byte[] test = new Byte[420000000];

I'm using ASP.NET 2.0. In ASP.Net 1.1 it works fine. So what am I doing
wrong?

What is the maximum length of a byte array?


Why you need such a big array in the first place?

I bet it depends of either (or both) IIS and the framework
 
"Ignacio Machin ( .NET/ C# MVP )" <ignacio.machin AT dot.state.fl.us> wrote
in message | Hi,
|
|
| | > Hi all,
| >
| > I'm getting an OutOfMemoryException when I initialize a byte array in
| > C# like this:
| >
| > Byte[] test = new Byte[420000000];
| >
| > I'm using ASP.NET 2.0. In ASP.Net 1.1 it works fine. So what am I doing
| > wrong?
| >
| > What is the maximum length of a byte array?
|
|
| Why you need such a big array in the first place?
|
| I bet it depends of either (or both) IIS and the framework
|


Hmm... IIS has nothing to do with this, ASP.NET runs in it's own process,
the framework only limit's the byte array size (just like any other object)
to ~2 GByte.
It's the way you allocate large contiguous blocks of memory from your
available virtual address space on a 32 bit OS which predicts the amount of
memory you can allocate. No single process, unless very carefully designed
and tested, will run for long before it encounters OOM exceptions when you
start allocating large objects or huge array's at wild. That's why asp.net
(the hosting process) provides the possibility to restart an AD and or the
worker process when a configurable memory threshold has been reached.

Willy.
 
Thanks for your answer!

I want to upload an image to a web application and therefore i need to
allocate this memory to fill the byte array with image data. Then I
need to do a resizing operation.
Hi,


Gerrit said:
Hi all,

I'm getting an OutOfMemoryException when I initialize a byte array in
C# like this:

Byte[] test = new Byte[420000000];

I'm using ASP.NET 2.0. In ASP.Net 1.1 it works fine. So what am I doing
wrong?

What is the maximum length of a byte array?


Why you need such a big array in the first place?

I bet it depends of either (or both) IIS and the framework

--
 
I think I'll try your advice at home where I have a 64Bit system but
with WinXP 32Bit OS installed. Do I also need to install a 64Bit OS to
try that ? (would be a task for the weekend ;-))
 
I think I'll try your advice at home where I have a 64Bit system but
with WinXP 32Bit OS installed. Do I also need to install a 64Bit OS to
try that ? (would be a task for the weekend ;-))


yes both, 64-Bit Windows (XP/2003/[Vista])
and the 64-Bit .NET Runtime.
For x64:
http://www.microsoft.com/downloads/details.aspx?familyid=B44A0000-ACF8-4FA1-AFFB-40E78D788B00

And compile for 'AnyCpu' target (or x64).

And maybe switching ASP.NET to 64-Bit
http://support.microsoft.com/kb/894435/en-us
 
Gerrit said:
I want to upload an image to a web application and therefore i need to
allocate this memory to fill the byte array with image data. Then I
need to do a resizing operation.

Is your image really 420MB large? That's really pretty massive. Can you
not do it in pieces? If not, perhaps you could launch a different
process to do the loading and resizing. That process could write the
result to disk, and then you could stream it to the web application.
 
Hi Jon,

yes this pictures are really up to 1GB. My method to shrink the images
works fine until a (decompressed) FileSize of 350 MB is reached.

If I do it in pieces, I also need to decompress the Picture on harddisk
as a bmp file and then read it into memory in small pieces. But to
decompress such a large picture with an .Net service or something i
will have to load it into memory anyway, which would cause a OOM
exception.

Do I have to use unmanaged code? Or is there a Windows API or anything
else I could use?

--
Gerrit Horeis

Software Developer
CI-Gate Development & Consulting GmbH
http://www.ci-gate.de
http://www.xira.de
http://www.bitbauer.de
 
As been said before in this thread, you won't be able to handle such huge
arrays on a 32 bit platform, not using unmanaged code or anything else.
You can't rely on the fact that such a huge free chunk of (CONTIGUOUS)
memory will ever be available, and if it's available at some point in time,
chances are that it's no longer available some time later as a result of
heap fragmentation.
A 32 bit process has a virtual memory space of max. ~GAB (or ~GAB for /GET
RAM tuned system), this space is shared by things like code modules, data
files, private heaps, process heaps the thread stacks etc...
.. After process creation and loading the initial modules (DLL's) this VIM
space is already fragmented, such that only ~1.GAB is available in one large
chunk (this figure is for the asp.net worker process).
But there is something else what scares me, when you are talking about
images and bitmaps in asp.net, does it mean that you are using
System.Drawing classes? If that's the case you should keep in mind that this
namespace is not supported (for good reasons) in asp nor in any other
service based application.

Willy.


| Hi Jon,
|
| yes this pictures are really up to 1GB. My method to shrink the images
| works fine until a (decompressed) FileSize of 350 MB is reached.
|
| If I do it in pieces, I also need to decompress the Picture on harddisk
| as a bmp file and then read it into memory in small pieces. But to
| decompress such a large picture with an .Net service or something i
| will have to load it into memory anyway, which would cause a OOM
| exception.
|
| Do I have to use unmanaged code? Or is there a Windows API or anything
| else I could use?
|
| --
| Gerrit Horeis
|
| Software Developer
| CI-Gate Development & Consulting GmbH
| http://www.ci-gate.de
| http://www.xira.de
| http://www.bitbauer.de
|
|
| Jon schrieb:
|
| > > I want to upload an image to a web application and therefore i need to
| > > allocate this memory to fill the byte array with image data. Then I
| > > need to do a resizing operation.
| >
| > Is your image really 420MB large? That's really pretty massive. Can you
| > not do it in pieces? If not, perhaps you could launch a different
| > process to do the loading and resizing. That process could write the
| > result to disk, and then you could stream it to the web application.
| >
| > --
| > Jon Skeet - <[email protected]>
| > http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
| > If replying to the group, please do not mail me too
|
 
Hi,

You will not be able to handle such a big image in memory. You would have to
devise a way to use it in parts.
Also select carefully your tools, I don't think System.Drawing can handle
such a big image, you would be better using DirectX or even better image
manipulation code written in unmanaged code.


--
--
Ignacio Machin,
ignacio.machin AT dot.state.fl.us
Florida Department Of Transportation

Gerrit said:
Hi Jon,

yes this pictures are really up to 1GB. My method to shrink the images
works fine until a (decompressed) FileSize of 350 MB is reached.

If I do it in pieces, I also need to decompress the Picture on harddisk
as a bmp file and then read it into memory in small pieces. But to
decompress such a large picture with an .Net service or something i
will have to load it into memory anyway, which would cause a OOM
exception.

Do I have to use unmanaged code? Or is there a Windows API or anything
else I could use?

--
Gerrit Horeis

Software Developer
CI-Gate Development & Consulting GmbH
http://www.ci-gate.de
http://www.xira.de
http://www.bitbauer.de
 
But I'm not able to handle a jpeg image (for example) in parts. If I
would do so, I would have load the whole image into memory to get parts
of it (because of compression)

I've decided to take a look on DirectX. Perhaps it has a method that
can handle jpeg images. If I use unmanaged code, I would have to write
the jpeg decompression algorithm on my own :-( This would be too hard
for me and too time intensive...

Thanks for all your help!

--
Gerrit Horeis

Software Developer
CI-Gate Development & Consulting GmbH
http://www.ci-gate.de
http://www.xira.de
http://www.bitbauer.de
Hi,

You will not be able to handle such a big image in memory. You would have to
devise a way to use it in parts.
Also select carefully your tools, I don't think System.Drawing can handle
such a big image, you would be better using DirectX or even better image
manipulation code written in unmanaged code.
 
Once again, you won't be able to handle such huge images (1GB) on Windows
32bit, whatever you try - managed code and GDI+, unmanaged code and GDI+ or
DirectX ... you name it - it will fail sooner than later, there is simply no
space in a process to accommodate such huge arrays, note I said arrays,
using System.Drawing(GDI+) you need memory space for the compressed raw
image (unmanaged heap), you need space in the managed heap for the image
data and you need space for the uncompressed image. Probably you want to
handle multiple images, so, even if it works for the first couple of images
(size dependent) if will certainly fail for one the following due to memory
fragmentation.

What I would suggest is that you write a very basic console application that
handles one image, run this in an unmanaged debugger like windbg.exe and
issue the !address -summary command. You will see that the largest free
region is ~1.1GB even before a single image is loaded.

Another point you seem to ignore, is the fact that this all doesn't work in
asp.net, you can't use GDI+ (System.Drawing) or DirectX in a web
application.


Willy.

| But I'm not able to handle a jpeg image (for example) in parts. If I
| would do so, I would have load the whole image into memory to get parts
| of it (because of compression)
|
| I've decided to take a look on DirectX. Perhaps it has a method that
| can handle jpeg images. If I use unmanaged code, I would have to write
| the jpeg decompression algorithm on my own :-( This would be too hard
| for me and too time intensive...
|
| Thanks for all your help!
|
| --
| Gerrit Horeis
|
| Software Developer
| CI-Gate Development & Consulting GmbH
| http://www.ci-gate.de
| http://www.xira.de
| http://www.bitbauer.de
|
| Ignacio Machin ( .NET/ C# MVP ) schrieb:
|
| > Hi,
| >
| > You will not be able to handle such a big image in memory. You would
have to
| > devise a way to use it in parts.
| > Also select carefully your tools, I don't think System.Drawing can
handle
| > such a big image, you would be better using DirectX or even better image
| > manipulation code written in unmanaged code.
| >
| >
| > --
| > --
| > Ignacio Machin,
| > ignacio.machin AT dot.state.fl.us
| > Florida Department Of Transportation
| >
| > | > > Hi Jon,
| > >
| > > yes this pictures are really up to 1GB. My method to shrink the images
| > > works fine until a (decompressed) FileSize of 350 MB is reached.
| > >
| > > If I do it in pieces, I also need to decompress the Picture on
harddisk
| > > as a bmp file and then read it into memory in small pieces. But to
| > > decompress such a large picture with an .Net service or something i
| > > will have to load it into memory anyway, which would cause a OOM
| > > exception.
| > >
| > > Do I have to use unmanaged code? Or is there a Windows API or anything
| > > else I could use?
| > >
| > > --
| > > Gerrit Horeis
| > >
| > > Software Developer
| > > CI-Gate Development & Consulting GmbH
| > > http://www.ci-gate.de
| > > http://www.xira.de
| > > http://www.bitbauer.de
| > >
| > >
| > > Jon schrieb:
| > >
| > >> > I want to upload an image to a web application and therefore i need
to
| > >> > allocate this memory to fill the byte array with image data. Then I
| > >> > need to do a resizing operation.
| > >>
| > >> Is your image really 420MB large? That's really pretty massive. Can
you
| > >> not do it in pieces? If not, perhaps you could launch a different
| > >> process to do the loading and resizing. That process could write the
| > >> result to disk, and then you could stream it to the web application.
| > >>
| > >> --
| > >> Jon Skeet - <[email protected]>
| > >> http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
| > >> If replying to the group, please do not mail me too
| > >
|
 
Hi,

It's been a while since I last saw the jpeg compression algorith but I'm
almost sure you can do it, it won't be efficient though. you would have to
discard parts of the image as you move forward (maybe saving chunk of it)

Again, 1 GB image is BIG, how many memory do you have to start with? And if
it's 1 GB in jpeg it would be mucho bigger in BMP !!!

And finally this is not task for a asp.net app at all.


What do you want to do with these images in the first place?
 
Ok I will use a service for the conversation in future, that's clear
now.


Its for an image management application. You can (or should be able to)
upload and download images up to 1GB in size (as spezified by the
customer). Supported filetypes should be every "usual" type you know
(therefore saving jpeg as bitmap is only a small part of all conversion
problems I'm running into. what is with png, Tiff, etc...they all can
be compressed)

Ok, Images > 500MB are very uncommon even in my case, and when somebody
wants to upload such an image, only bitmaps and tiffs would be
allowed.(By the way: does anybody know a way to determine filetype and
size BEFORE uploading?)

I've found a C++ dll which I would like to give a try to. Its called
FreeImage
http://freeimage.sourceforge.net/

Another thing would be to setup a Windows XP 64 Bit OS this
weekend...perhaps this helps too...

--
Gerrit Horeis

Software Developer
CI-Gate Development & Consulting GmbH
http://www.ci-gate.de
http://www.xira.de
http://www.bitbauer.de
 
Setting Up a Windows XP 64 works really better with large images! I was
able to load a a 400 MB tif File into memory (on my 1GB RAM, 2800+ AMD
PC).
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Back
Top