Large image load error that do work in C++

[Guest]

I’m wrote a small DLL that used the FreeImage.DLL (that can be found at
http://www.codeproject.com/bitmap/graphicsuite.asp).
I also wrote a small console application in C++ (unmanaged) that uses the
DLL above.

Now the application, together with the above DLL’s is successfully loading a
TIF image file (62992 x 113386 Pixels, Huffman RLE compression, 3200 x 3200
DPI resolution, binary colored (1 Bit Per Pixel), file on disk size 43.08 MB
(45,169,042 Bytes). It should consume 851.66 MB of memory when loaded).

Now I wrote a simple C# application that used that same DLL’s (that are
written in C, C++) and that do the same thins as the C++ application.
But the C# application is failing to load the image (FreeImage_Load()
returns null).

The C# application itself is taking about 8MB and the loaded image loading
takes 851.66 MB, so it means that is should take no more then 860 MB of
memory all together.
The PC is installed with 4 GB RAM, and the WinXP Pro is only seeing 3.25 GB.
I also set the Boot.ini with /3GB flag and my C# application is liked with
the IMAGE_FILE_LARGE_ADDRESS_AWARE flag.
I do not the 3GB for my app is needed, but what the heck.

It looks to me like a bug in the .NET

Can anybody tell my how to solve that?

 

[Nicholas Paldino [.NET/C# MVP]]

Sharon,

Can you show the unmanaged declaration (along with all of the supporting
structures, if necessary) and the corresponding managed declarations you
have? It sounds like you arent marshaling something correctly.

 

[Willy Denoyette [MVP]]

You have been asking this same question here and in other NG's and forums .
Well, I'm sorry, but the answer is still the same. The part above 3.25 GB,
is taken by the hardware components like Disk adapters, Video adapters (you
even mentioned that you have two of them installed) etc... that have memory
mapped into the 32 bit address space of Windows, that means that the OS
itself sees (and uses) the memory but it's not available for "user"
software, that is why it doesn't show up.
There is nothing you can do about this, other than switching to XP 64 bit
and hope that you have a System that makes it possible to remap the
controler memory.

I also told you before that you can't expect to have a "single contigious
block of 800MB free memory" available in a Win32 process, this is due to the
process heap fragmentation by data and code modules (DLL's).
What you are trying now won't work, simply because by creating a C#
application you need to load additional modules like the CLR modules, the
BCL and probably Windows.Forms and Windows.Drawing modules which results in
a smaller (large) block of free memory available for your image data, than
you had available in your simple C++ console program (which by the way
doesn't use GDI+).
Again, you have two options, write all of your code in C++ (but don't use
GDI+) or move to a 64bit OS.


Willy.

 

[TerryFei]

Hi Sharon,

I also suggest you can try to migration the code to 64bit OS(for example
Windows XP Professional 64-bit Edition), if it's possible for you.
64bit OS(currently supports 32 GB RAM and has a theoretical limit of around
16 TB RAM. This dwarfs the 2-3 GB limits on 32-bit systems that are
rapidly becoming bottlenecks to increasingly complex software applications.
32-bit applications are not capable of supporting many customers' growing
memory requirements. For those that haven't reached the limit yet (i.e.
working with high-resolution images or large amounts of warehoused data),
roughly exponential growth in resource requirements means that they will
reach these limits soon. So I think 64bit OS is a better solution for you.

Best Regards,

Terry Fei[MSFT]
Microsoft Community Support
Get Secure! www.microsoft.com/security
(This posting is provided "AS IS", with no warranties, and confers no
rights.)

 

[Guest]

I am asking similar question, but it is no the same.
This time I wrote a C++ library that uses the GDI+. But when I use this DLL
by C# application, it works fine. But when I use this same DLL on the same
PC, it fails.
The C# application is using only 8BM before accessing the C++ DLL, that
means that it consume less then 900 MB when invoking the DLL function the
load the image.
850MB of continues memory should be no trouble to allocate even on a 2GB RAM
PC, the C++ application proves that when it successfully loading that same
image.

The difference in memory consumption between the C++ application and the C#
application is just 6MB.

We can not move to 64bit OS at this point, we simply can not.
I also can not write the whole application in C++, because this is a large
application that is almost done, and we do not have the time to re write in
C++.

I’m very much thankful for the help you gave me in all your kind answers,
and because of this answers I’m trying to write a GUI image control from
scratch.
First I’m will write a Win32 control in C++, and then I will write the .NET
control that will wrap the C++ control.

I’m sure that some kind of solution can be found, otherwise it simply a bug
in the .NET.

 

[Guest]

We can not move to 64bit OS at this point, we simply can not.
I also can not write the whole application in C++, because this is a large
application that is almost done, and we do not have the time to re write in
C++.

I wrote a C++ library that uses the GDI+. But when I use this DLL by C#
application, it works fine. But when I use this same DLL on the same PC, it
fails.
The C# application is using only 8BM before accessing the C++ DLL, that
means that it consume less then 900 MB when invoking the DLL function the
load the image.
850MB of continues memory should be no trouble to allocate even on a 2GB RAM
PC, the C++ application proves that when it successfully loading that same
image.

The difference in memory consumption between the C++ application and the C#
application is just 6MB.

I’m sure that some kind of solution can be found, otherwise it simply a bug
in the .NET.

 

[Guest]

The unmanaged C++ library code:
#ifdef LARGEIMAGELOADER_EXPORTS
#define LARGEIMAGELOADER_API __declspec(dllexport)
#else
#define LARGEIMAGELOADER_API __declspec(dllimport)
#endif

// Release the image resources.
LARGEIMAGELOADER_API void FreeImage()
{
if( handleFI_MP != 0 )
{
FreeImage_CloseMultiBitmap(handleFI_MP);
}
else if( handleFI != 0 )
{
FreeImage_Unload(handleFI);
}
handleFI = 0;
handleFI_MP = 0;
}

LARGEIMAGELOADER_API BYTE* GetImageByteArray(char* imageFileName)
{
// load plugins
FreeImage_DeInitialise();
FreeImage_Initialise(FALSE);

// first, release previous allocated image
FreeImage();

char szLongPath[2*_MAX_PATH] = { 0 };
BOOL bSucees = FALSE;
int pageCount = 1;
handleFI = 0;
handleFI_MP = 0;
FREE_IMAGE_FORMAT fif;

fif = FreeImage_GetFIFFromFilename(imageFileName);
if( fif == FIF_UNKNOWN )
{
FreeImage();
return NULL;
}
else
{
handleFI_MP = FreeImage_OpenMultiBitmap(fif, imageFileName, FALSE, FALSE,
FALSE);
if( handleFI_MP )
{
pageCount = FreeImage_GetPageCount(handleFI_MP);
if( pageCount <= 1 )
{
FreeImage_CloseMultiBitmap(handleFI_MP);
handleFI_MP = NULL;
}
}
if( ! handleFI_MP )
{
// This is the point of failure when used by the C# application.
handleFI = FreeImage_Load(fif, imageFileName);
}
}
bSucees = (handleFI!=NULL) || (handleFI_MP!=NULL);

// failure check
if( fif == FIF_UNKNOWN || ! bSucees )
{
return NULL;
}

BYTE* pData = FreeImage_GetBits(handleFI);
return pData;
}

The C++ application that successfully uses the unmanaged C++ library above:

int _tmain(int argc, _TCHAR* argv[])
{
BYTE* pImg = GetImageByteArray("TheImageFile.tif");
FreeImage();
return 0;
}

The C# application that fail to use the unmanaged C++ library above:

unsafe
{
byte* pData;
string fileName = "TheImageFile.tif";
System.Text.ASCIIEncoding ascii = new System.Text.ASCIIEncoding();
byte[] bytes = ascii.GetBytes(fileName);
fixed( byte* chFileName = bytes )
{
// It fail in here.
pData = GetImageByteArray(chFileName);
}
FreeImage();
}


I hope it helps.

 

[Guest]

I forgot the dll imports PInvoke that I use in the C# application, so here it
is:

DllImport("LargeImageLoader.dll")]
private static extern void FreeImage();

[DllImport("LargeImageLoader.dll")]
private static extern unsafe byte* GetImageByteArray(byte* imageFileName);


I hope it helps to find a solution.

 

[Willy Denoyette [MVP]]

I am asking similar question, but it is no the same.
This time I wrote a C++ library that uses the GDI+. But when I use this
DLL
by C# application, it works fine. But when I use this same DLL on the same
PC, it fails.
The C# application is using only 8BM before accessing the C++ DLL, that
means that it consume less then 900 MB when invoking the DLL function the
load the image.
850MB of continues memory should be no trouble to allocate even on a 2GB
RAM
PC, the C++ application proves that when it successfully loading that same
image.

The difference in memory consumption between the C++ application and the
C#
application is just 6MB.

We can not move to 64bit OS at this point, we simply can not.
I also can not write the whole application in C++, because this is a large
application that is almost done, and we do not have the time to re write
in
C++.

I’m very much thankful for the help you gave me in all your kind
answers,
and because of this answers I’m trying to write a GUI image control from
scratch.
First I’m will write a Win32 control in C++, and then I will write the
.NET
control that will wrap the C++ control.

I’m sure that some kind of solution can be found, otherwise it simply a
bug
in the .NET.

1. Your C# program should not use unsafe blocks.
Use a string as argument to GetImageByteArray and an IntPtr as return.


[DllImport("LiLoader.dll")]
private static extern void FreeImageEx();
[DllImport("LiLoader.dll")]
private static extern IntPtr GetImageByteArray(string imageFileName);

static void Main()
{
IntPtr pData = GetImageByteArray("TheImageFile.tif");
if(pData != IntPtr.Zero)
{
// Do something with with ethe IntPtr value returned...
}
FreeImage();
}

I did run this with the C++ code you posted previously (and FreeImage dll),
all went fine (tough I don't think the code does anything significantly).

2. The C++ library (FreeImage) you are talking about does not use GDI+, but
I guess this is what you meant.
3. The pure C++ solution has a contiguous free space area of 1.621GB when
started.
4. The C# code using the same library (FreeImage) and your C++ wrapper code
has a largest free block of ~1.406.000Kb out of a total of ~2.013.000 Kb
free memory (the second largest block being ~350 MB).

But, your simple C# code is a do-nothing (the GC heap is almost empty), that
is the GetImageByteArray reads the image (not sure though) and returns a
pointer to a byte buffer in unmanaged heap memory, so far so good this
should work for your 850MB image file. But the problem is that you C# code
is a console application, that means Windows.Forms is not loaded; and more
importantly Windows.Drawing in not loaded (this includes the GDI+ native
library used by the BitMap stuff).
Now, if you say you don't need System.WinForms and System.Drawing in your C#
(or managed) application, then and only then this should work, but once you
bring in System.Drawing (GDI+) it might fail like it did before. The reason
is that the gdi+ library (gdiplus.dll) gets mapped somewhere at location
0x4ec50000 in your VA further reducing the largest area of free memory which
starts at 0x 0x03000000 to something like 1.183 Gb (0x4ec50000 -
0x03000000).
If from this 1.183GB you already consumed 850MG, there is only 333MB left
for the GC heap, if your managed application can work with this and the
other large free block in top of memory, all if fine, but don't be
suppressed when someone spoils the party and further fragments the heap
during run-time. Especially xpsp2res.dll loaded by some programs is a killer
(see http://support.microsoft.com/?kbid=894472 for details and a hotfix).

Note that all I'm telling you can easily be watched at run-time if you start
your application with native code debugging enabled.


Willy.

 

[Guest]

Hi Willy,

I have noticed some mistakes in my latest post. The fix is:
The C++ library does no use the GDI+. When I use this DLL by the C#
application, it does NOT works (it fail inside the GetImageByteArray() at
line FreeImage_GetBits(handleFI) that returns NULL each time).
But when I use this same DLL on the same PC with the C++ application, it
works fine.

The C# application is not a console application, it does use Windows.Forms
and Windows..Drawing.
[End of fix]

1. I changed the C# code as you suggested.
Can you tell me how to use the IntPtr to convert it to byte[] ?

2. You are right.

3. + 4. Good enough.

I understand that it might fail in a Windows.Form application, but it looks
like that it may fail statistically and not 100% of the times (even you say
“might failâ€), which is the case.
But event so, it look like that the .NET framework memory fragmentation is
very poor (I understand that the .NET framework v1.1 service pack try to fix
that and that the v2.0 makes it even worse).
“xpsp2res.dll loaded by some programs is a killerâ€, yes, I must agree that
there are real killers in the .NET framework.

I have heard that when the unmanaged C++ library is allocating the memory
for the image loading, and returns a pointer to the managed code, the PInvoke
marshal copies the this unmanaged memory to the managed memory, and therefore
uses twice the memory (2 x 850 MB = 1.6GB), that will fail in large memory
allocation like in my case.
Is that true? Should I expect one more failure after overcoming the current
one?

 

[Willy Denoyette [MVP]]

Sharon,

See inline.

Willy.

Hi Willy,

I have noticed some mistakes in my latest post. The fix is:
The C++ library does no use the GDI+. When I use this DLL by the C#
application, it does NOT works (it fail inside the GetImageByteArray() at
line FreeImage_GetBits(handleFI) that returns NULL each time).
But when I use this same DLL on the same PC with the C++ application, it
works fine.

The C# application is not a console application, it does use Windows.Forms
and Windows..Drawing.
[End of fix]

1. I changed the C# code as you suggested.
Can you tell me how to use the IntPtr to convert it to byte[] ?

Ok, here we go again, you have a contigious block of 850MB of image data
allocated in unmanaged memory (allocated by the unmanaged module FreeImage
dll), right?
And (as I did expect )) you want this data to be available in managed
memory (the GC heap)
, well, you can copy this IntPtr to a byte[] (this will end in the the Large
Object Heap) by using Marshal.Copy(...), or you can pick the pointer from
the IntPtr and use this one in unsafe code blocks to copy the contents to a
managed byte[](using pointers).
BUT.... to do this you'll need another 850MB "contigious block" of free
memory, which you don't have available in a 32bit process running on
Windows. This has NOTHING to do with .NET or .NET's memory fragmentation,
the same will fail in an unmanaged C++ application if you try to copy 850MB
of data from one location to another loaction in your process address space.
(The largest contigious block of memory in a 32bit process is 1.6GB, even
with 4GT tuning enabled!!!!!).
Now, what you can to do is access the unmanaged buffer using unsafe code and
pointers that directly access the buffer, but all depends on what you are
trying to achieve, and that is the question which remains unanswered - What
exactly are you expecting to do with the data in the buffer in a C#
application? Why exactly do you need this to be written in C#? Your problem
can be solved if you write this in unmanaged C++ (provided you don't need to
copy the buffer), or if you make sure you don't copy the buffer to managed
memory when using C#, but the question is why would you use C# then?.

2. You are right.

3. + 4. Good enough.

NO, not if you want the unmanaged block in a managed byte[].

I understand that it might fail in a Windows.Form application, but it
looks
like that it may fail statistically and not 100% of the times (even you
say
“might failâ€), which is the case.
But event so, it look like that the .NET framework memory fragmentation is
very poor (I understand that the .NET framework v1.1 service pack try to
fix
that and that the v2.0 makes it even worse).

Wrong see above.

“xpsp2res.dll loaded by some programs is a killerâ€, yes, I must agree
that
there are real killers in the .NET framework.

Please don't blaim .NET, xpsp2res.dll is not part of .NET, it's part of the
SP2 service pack and there is a fix, all this is clearly explained in the KB
article.

I have heard that when the unmanaged C++ library is allocating the memory
for the image loading, and returns a pointer to the managed code, the
PInvoke
marshal copies the this unmanaged memory to the managed memory, and
therefore
uses twice the memory (2 x 850 MB = 1.6GB), that will fail in large memory
allocation like in my case.

No it only returns a pointer (wrapped by the IntPtr), it's only when you
want that block of data in your managed (GC collected heap) that it should
be marshaled (copied).
small nitpick, 2 * 850 = 1.782 KB.

Is that true? Should I expect one more failure after overcoming the
current
one?

Probably, as I told you many times before.

 

[Guest]

Hi Willy,

Thanks again for your help.

It will be good if the allocated memory for the image loading will be
available in the C# app, so I thing I will work with unsafe code and pointers
to do it.

I do not have the xpsp2res.dll problem because unmanaged C++ application
successfully using large memory chunk (850 MB loaded image).

What I’m trying to achieve?
Ok, we are working on a project that among other think receives on run time
large reference image on file, this image is then loaded to memory and
scanned to compare it with a smaller images generated on run time using
cameras.
One of my role in this project is to create GUI that will also has a control
that shows the reference image and draw on it some more data and has a
zooming and scrolling capabilities with high resolution.

As we already know, the .NET PictureBox control can not handle that big
image. So I need to build one from scratch. But then again, I fail to load
this image at all in the C# application.

So I’m trying to find some reasonable way to create this king of control.
As you know now, I’m trying to approach this issue from many ways; the last
resort that I can think of is divide the project into two processes (at least
two), managed and unmanaged. The unmanaged process will load the image and
will help with the zooming and scrolling by each time copying some portion of
the loaded image and use it to create a new image that will have the size of
the GUI control client area. Then the C# process will take this new smaller
image and will draw it on the control.
The trouble with this solution is it requires more then one process, some
kind of process communication (COM+ or shared memory etc.), maybe some
reduced performance, longer development time that we do not have and who
knows what more.

It should be much simpler.

Another bad solution will be to create a smaller image (a lot smaller that
will result on poor summing resolution) from the large reference image (that
also I need an unmanaged process for it), and use it in the .NET PictureBox
control.
But for that I need to do the resizing in run time that takes a lot of
memory, and I will need to hold in memory 2 images, the original one (for
algorithm issues) and another one for the GUI.

We can not move the entire project to C++ because this is a big project,
most of it is already written, and it will consume a long time to rewrite it,
time that we do not have.

I hope I made the problem more understood.
My problem with the .NET frame work is that event if a .NET application can
have only ~1.406.000Kb minus all the needed memory for the .NET should leave
me with a least 850MB of continues memory.
The difference between C++ application and a C# application is too great
concerning memory consumption and / or fragmentation.

 

[Willy Denoyette [MVP]]

Sharon,

Inline.

Willy.

Hi Willy,

Thanks again for your help.

It will be good if the allocated memory for the image loading will be
available in the C# app, so I thing I will work with unsafe code and
pointers
to do it.

I do not have the xpsp2res.dll problem because unmanaged C++ application
successfully using large memory chunk (850 MB loaded image).

Well, actually an unmanaged C++ application using graphics (GDI etc.) might
load xpsp2res.dll, so any managed app. that interops with MFC C++ will have
this module mapped in it's address space. I'm pretty sure GraphicsSuite
(http://www.codeproject.com/bitmap/graphicsuite.asp).) do load xpsp2res.dll
(note that this is for XP/SP2 only).
It seems like you do not correctly understand what .NET is (I'm not talking
about the marketing term), it's nothing more than an OO oriented API
supported by a sophisticated run-time (the CLR). The API is thin, in some
area's almost nothing more than a direct call into unmanaged code loaded by
the CLR or a call into the CLR itself. The bulk of the code running in a
process is nothing more than unmanaged code taken from the loaded win32
DLL's available in the system (not .NET). Even the "Fatter" managed API's
like Windows.Forms are simply wrappers arround unmanaged Windows code used
by any other unmnaged process/windows application. That means that most
restrictions like the memory usage pattern you are describing, is also valid
for non ".NET" applications.

What I’m trying to achieve?
Ok, we are working on a project that among other think receives on run
time
large reference image on file, this image is then loaded to memory and
scanned to compare it with a smaller images generated on run time using
cameras.
One of my role in this project is to create GUI that will also has a
control
that shows the reference image and draw on it some more data and has a
zooming and scrolling capabilities with high resolution.

As we already know, the .NET PictureBox control can not handle that big
image. So I need to build one from scratch. But then again, I fail to load
this image at all in the C# application.

Beware, there is no such thing as a .NET PictureBox, the PictureBox is an
unmanaged control that uses GDI, not GDI+) , just the same control as would
be used by any unmanaged application. But as you already experienced, the PB
was not meant to be used to hold images that are 10 times the sceen size,
that's one thing, the other is that the PB can handle images as large as the
amount of free memory (contigious) allows.

So I’m trying to find some reasonable way to create this king of
control.
As you know now, I’m trying to approach this issue from many ways; the
last
resort that I can think of is divide the project into two processes (at
least
two), managed and unmanaged. The unmanaged process will load the image and
will help with the zooming and scrolling by each time copying some portion
of
the loaded image and use it to create a new image that will have the size
of
the GUI control client area. Then the C# process will take this new
smaller
image and will draw it on the control.

Waw, this will be slow!

The trouble with this solution is it requires more then one process, some
kind of process communication (COM+ or shared memory etc.), maybe some
reduced performance, longer development time that we do not have and who
knows what more.

If there is a way to go, it's the Shared memory way you should go.

It should be much simpler.

Yes, and that's when you are writing the whole image handling stuff in C++,
or move to 64bit.

Another bad solution will be to create a smaller image (a lot smaller that
will result on poor summing resolution) from the large reference image
(that
also I need an unmanaged process for it), and use it in the .NET
PictureBox
control.
But for that I need to do the resizing in run time that takes a lot of
memory, and I will need to hold in memory 2 images, the original one (for
algorithm issues) and another one for the GUI.

We can not move the entire project to C++ because this is a big project,
most of it is already written, and it will consume a long time to rewrite
it,
time that we do not have.

I hope I made the problem more understood.

More or less, but I don't get what you mean with a large project, no-one
said you can't mix C++ and managed code in a large project, all you have to
do is take the appropriate language for the imaging stuff .

My problem with the .NET frame work is that event if a .NET application
can
have only ~1.406.000Kb minus all the needed memory for the .NET should
leave
me with a least 850MB of continues memory.

No, no 1.4GB is the max. free block when GDI+ is not loaded into the
process, but if you use C# with System.Drawing, you bring in GDI+, the same
is true for C++, once you load gdiplus.dll in a process the largest free
block is reduced to 1.83GB, because the dll is loaded at baseaddress
0x5ec50000. So, yes probaly 850MB will not be a problem as long as you don't
have to copy this whole block to the managed heap.

The difference between C++ application and a C# application is too great
concerning memory consumption and / or fragmentation.

How and what do you compare? The fragmentation is the same, did you actually
run the C# code that I corrected, it should work (as long as you don't copy
the buffer, but that's the same for the C++ version)?