large array is not garbage collected

[Howard Weiss]

I am reading a 160 mB file in my application. I want to access the file as
shorts. FileStream->Read wants to access the file as Byte so I do the
following

1) Read the filesize
2) Create a Byte[] array of the appropriate size
3) Read the file
4) Create a (global) nogc array of the same size and copy the data from the
Byte array to the gc array
5) Delete the Byte[] array

as follows:

FileStream *myFile = new FileStream(pPath, FileMode::Open,
FileAccess::Read);
__int64 myFileSize = myFile->get_Length();
Byte[] myFileData = new Byte[(int) myFileSize];
myFile->Read(myFileData, 0, (int) myFileSize);
int x = myFileData->Length;
pFileData = new unsigned char [(int) myFileSize];
for (int ii = 0; ii < myFileSize; ii++)
pFileData[ii] = myFileData[ii];
delete myFileData;
myFile->Close();

The code for delete myFileData sets the pointer to 0. Presumeably, this
will cause the garbage collector to reclaim the memory. Using the task
manager, I observe that the Mem Usage for my process increases by twice the
file size (over 300 mB) and never decreases (e.g. the Byte array is never
garbage collected). I have tried adding a GC::Collect() call after the
delete myFileData but nothing happens.

Any suggestions?

Howard Weiss

 

[Howard Weiss]

Correction: The 4th line is

Byte myFileData[] = new Byte[(int)myFileSize];

 

[Ross Jones]

Howard,

I'm no expert but it looks to me like you have two arrays of size
myFileSize and I was under the
impression that the CLR did not bother releasing memory back to the OS
unless it was explicitly
required.

I'm sorry I haven't answered your question but seeing the 160Mb file
prompted me to ask why you are not mapping a view of the file instead of
loading it into arrays?

Ross.

I am reading a 160 mB file in my application. I want to access the file as
shorts. FileStream->Read wants to access the file as Byte so I do the
following

1) Read the filesize
2) Create a Byte[] array of the appropriate size
3) Read the file
4) Create a (global) nogc array of the same size and copy the data from the
Byte array to the gc array
5) Delete the Byte[] array

as follows:

FileStream *myFile = new FileStream(pPath, FileMode::Open,
FileAccess::Read);
__int64 myFileSize = myFile->get_Length();
Byte[] myFileData = new Byte[(int) myFileSize];
myFile->Read(myFileData, 0, (int) myFileSize);
int x = myFileData->Length;
pFileData = new unsigned char [(int) myFileSize];
for (int ii = 0; ii < myFileSize; ii++)
pFileData[ii] = myFileData[ii];
delete myFileData;
myFile->Close();

The code for delete myFileData sets the pointer to 0. Presumeably, this
will cause the garbage collector to reclaim the memory. Using the task
manager, I observe that the Mem Usage for my process increases by twice the
file size (over 300 mB) and never decreases (e.g. the Byte array is never
garbage collected). I have tried adding a GC::Collect() call after the
delete myFileData but nothing happens.

Any suggestions?

Howard Weiss

 

[Howard Weiss]

Ross -

I agree that it is somewhat short sighted to have 2 full copies of the array
in memory at the same time. Nonetheless, I would have expected to see my
memory usage increase by twice the file size and then collapse back to 1x
the file size.

Please point me to an appropriate class to map a view of the data

Thanks

Howard Weiss

Howard,

I'm no expert but it looks to me like you have two arrays of size
myFileSize and I was under the
impression that the CLR did not bother releasing memory back to the OS
unless it was explicitly
required.

I'm sorry I haven't answered your question but seeing the 160Mb file
prompted me to ask why you are not mapping a view of the file instead of
loading it into arrays?

Ross.

"Howard Weiss" wrote file

as

shorts. FileStream->Read wants to access the file as Byte so I do the
following

1) Read the filesize
2) Create a Byte[] array of the appropriate size
3) Read the file
4) Create a (global) nogc array of the same size and copy the data

from
the

Byte array to the gc array
5) Delete the Byte[] array

as follows:

FileStream *myFile = new FileStream(pPath, FileMode::Open,
FileAccess::Read);
__int64 myFileSize = myFile->get_Length();
Byte[] myFileData = new Byte[(int) myFileSize];
myFile->Read(myFileData, 0, (int) myFileSize);
int x = myFileData->Length;
pFileData = new unsigned char [(int) myFileSize];
for (int ii = 0; ii < myFileSize; ii++)
pFileData[ii] = myFileData[ii];
delete myFileData;
myFile->Close();

The code for delete myFileData sets the pointer to 0. Presumeably, this
will cause the garbage collector to reclaim the memory. Using the task
manager, I observe that the Mem Usage for my process increases by

twice
the

file size (over 300 mB) and never decreases (e.g. the Byte array is never
garbage collected). I have tried adding a GC::Collect() call after the
delete myFileData but nothing happens.

Any suggestions?

Howard Weiss

 

[Ross Jones]

Ah. Um. I don't think there is a managed class provided by Microsoft to do
this (although I could be wrong).
It might be best to wrap some P/Invoke calls in a wrapper class and then use
the Win32 MapViewOfFile to access the file/memory in chunks.

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/base/mapviewoffile.asp

I agree with the expected behaviour you saw, and I was disappointed when I
saw C# doesn't do it either. But better minds than mine tell me that it's
better that way

Ross.

Ross -

I agree that it is somewhat short sighted to have 2 full copies of the array
in memory at the same time. Nonetheless, I would have expected to see my
memory usage increase by twice the file size and then collapse back to 1x
the file size.

Please point me to an appropriate class to map a view of the data

Thanks

Howard Weiss

Howard,

I'm no expert but it looks to me like you have two arrays of size
myFileSize and I was under the
impression that the CLR did not bother releasing memory back to the OS
unless it was explicitly
required.

I'm sorry I haven't answered your question but seeing the 160Mb file
prompted me to ask why you are not mapping a view of the file instead of
loading it into arrays?

Ross.

I am reading a 160 mB file in my application. I want to access the

file

as
shorts. FileStream->Read wants to access the file as Byte so I do the
following

1) Read the filesize
2) Create a Byte[] array of the appropriate size
3) Read the file
4) Create a (global) nogc array of the same size and copy the data from
the
Byte array to the gc array
5) Delete the Byte[] array

as follows:

FileStream *myFile = new FileStream(pPath, FileMode::Open,
FileAccess::Read);
__int64 myFileSize = myFile->get_Length();
Byte[] myFileData = new Byte[(int) myFileSize];
myFile->Read(myFileData, 0, (int) myFileSize);
int x = myFileData->Length;
pFileData = new unsigned char [(int) myFileSize];
for (int ii = 0; ii < myFileSize; ii++)
pFileData[ii] = myFileData[ii];
delete myFileData;
myFile->Close();

The code for delete myFileData sets the pointer to 0. Presumeably, this
will cause the garbage collector to reclaim the memory. Using the task
manager, I observe that the Mem Usage for my process increases by twice
the
file size (over 300 mB) and never decreases (e.g. the Byte array is never
garbage collected). I have tried adding a GC::Collect() call after the
delete myFileData but nothing happens.

Any suggestions?

Howard Weiss

 

[Ben Schwehn]

Any suggestions?


make sure you use the release build config

ben

 

[Vladimir Nesterovsky]

Hello,

Byte myFileData[] = new Byte[(int)myFileSize];

Did you mean:

Byte[] myFileData = __gc new Byte[(int) myFileSize];