OutOfMemoryException When Loading Huge file into MemoryStream

G

Guest

Hello,

I am the sample FPSEPublish
(http://blog.baeke.info/blog/_archives/2005/3/3/393158.html) code to upload a
document to Sharepoint (WSS). This works perfectly for samll documents.

Problem:

When I attempt to upload a huge document (300Megabayte) on a PC with 3.6Gig
RAM I am getting a OutOfMemoryException when the program attempts to read the
file into a MemoryStream.

The problem has nothing to do with Sharepoint as it is happening prior to
the FP RPC upload atempt.

Questions: Is there a limit on the amount of memory that is assigned to the
GarbageCollected heap that is being reached? Using the Sysinternals
ProcessExplorer program I can see the amount of memory being consumed by the
process as follows:

Total Reserved Bytes: 897,544,192

I have tried to understand how the CLR is allocating memory to no avail.

Any help/pointer is greatly appreciated

Sample Code:

+++++++++++++++++++++++++++++++++++
public void PutDocument(string uri, string fileName, string metaInfo)
{
//GC.Collect();
//GC.WaitForPendingFinalizers();
//GC.Collect();


Uri myUri = new Uri(uri);
string webUrl, fileUrl;
UrlToWebUrl(uri, out webUrl, out fileUrl);

if (null == metaInfo)
metaInfo = "";

if (!File.Exists(fileName))
throw new Exception("Could not find file" + fileName);

string postBody = String.Format
"method=put+document&service_name=&document=[document_name={0};meta_info=[{1}]]&put_option=overwrite&comment=&keep_checked_out=false\n",
HttpUtility.UrlEncode(fileUrl),
metaInfo);

ASCIIEncoding encoding = new ASCIIEncoding();
MemoryStream stream = new MemoryStream();
stream.Write(encoding.GetBytes(postBody), 0, postBody.Length);

FileStream fs = File.OpenRead(fileName);
byte[] b = new byte[4096];
while (fs.Read(b, 0, b.Length) > 0)
{
//GC.Collect();
//GC.WaitForPendingFinalizers();
//GC.Collect();

try
{
stream.Write(b, 0, b.Length); //FAILS TO WRITE TO STREAM AFTER STREAM
SIZE REACHES APPROX 270Meg
//System.Diagnostics.Debug.WriteLine(stream.Length.ToString());
}
catch(Exception ex)
{
System.Diagnostics.Trace.WriteLine(ex.Message);
}
}
fs.Close();

SendRequest(myUri.GetLeftPart(UriPartial.Authority) + webUrl +
"/_vti_bin/_vti_aut/author.dll", stream.GetBuffer(), stream.Length);
stream.Close();
}


+++++++++++++++++++++++++++++++++++
 
W

Willy Denoyette [MVP]

| Hello,
|
| I am the sample FPSEPublish
| (http://blog.baeke.info/blog/_archives/2005/3/3/393158.html) code to
upload a
| document to Sharepoint (WSS). This works perfectly for samll documents.
|
| Problem:
|
| When I attempt to upload a huge document (300Megabayte) on a PC with
3.6Gig
| RAM I am getting a OutOfMemoryException when the program attempts to read
the
| file into a MemoryStream.
|
| The problem has nothing to do with Sharepoint as it is happening prior to
| the FP RPC upload atempt.
|
| Questions: Is there a limit on the amount of memory that is assigned to
the
| GarbageCollected heap that is being reached? Using the Sysinternals
| ProcessExplorer program I can see the amount of memory being consumed by
the
| process as follows:
|
| Total Reserved Bytes: 897,544,192
|
| I have tried to understand how the CLR is allocating memory to no avail.
|
| Any help/pointer is greatly appreciated
|
| Sample Code:
|
| +++++++++++++++++++++++++++++++++++
| public void PutDocument(string uri, string fileName, string metaInfo)
| {
| //GC.Collect();
| //GC.WaitForPendingFinalizers();
| //GC.Collect();
|
|
| Uri myUri = new Uri(uri);
| string webUrl, fileUrl;
| UrlToWebUrl(uri, out webUrl, out fileUrl);
|
| if (null == metaInfo)
| metaInfo = "";
|
| if (!File.Exists(fileName))
| throw new Exception("Could not find file" + fileName);
|
| string postBody = String.Format(
|
"method=put+document&service_name=&document=[document_name={0};meta_info=[{1}]]&put_option=overwrite&comment=&keep_checked_out=false\n",
| HttpUtility.UrlEncode(fileUrl),
| metaInfo);
|
| ASCIIEncoding encoding = new ASCIIEncoding();
| MemoryStream stream = new MemoryStream();
| stream.Write(encoding.GetBytes(postBody), 0, postBody.Length);
|
| FileStream fs = File.OpenRead(fileName);
| byte[] b = new byte[4096];
| while (fs.Read(b, 0, b.Length) > 0)
| {
| //GC.Collect();
| //GC.WaitForPendingFinalizers();
| //GC.Collect();
|
| try
| {
| stream.Write(b, 0, b.Length); //FAILS TO WRITE TO STREAM AFTER STREAM
| SIZE REACHES APPROX 270Meg
| //System.Diagnostics.Debug.WriteLine(stream.Length.ToString());
| }
| catch(Exception ex)
| {
| System.Diagnostics.Trace.WriteLine(ex.Message);
| }
| }
| fs.Close();
|
| SendRequest(myUri.GetLeftPart(UriPartial.Authority) + webUrl +
| "/_vti_bin/_vti_aut/author.dll", stream.GetBuffer(), stream.Length);
| stream.Close();
| }
|
|
| +++++++++++++++++++++++++++++++++++

|
| Total Reserved Bytes: 897,544,192

This doesn't mean a lot, but is this the reserved bytes before you attempted
the load?
Anyway, the exception means that you don't have a free contigious area of
270MB available in your process 2GB Virtual address space.
Also keep in mind that strings in .NET are really System.Char arrays, where
a char is 16 bit. So if your doc is ASCII encoded, it will take twice the
size of the file in memory.

Willy.
 
J

Jon Skeet [C# MVP]

Also keep in mind that strings in .NET are really System.Char arrays, where
a char is 16 bit. So if your doc is ASCII encoded, it will take twice the
size of the file in memory.

Where is the file being treated as a string? The ASCII encoding is only
used for the URL, as far as I can see.
 
J

Jon Skeet [C# MVP]

Naamat said:
I am the sample FPSEPublish
(http://blog.baeke.info/blog/_archives/2005/3/3/393158.html) code to upload a
document to Sharepoint (WSS). This works perfectly for samll documents.

No it doesn't, unfortunately.

Aside from the memory problems you're getting, the code is actually
wrong, and stylistically flawed too (I'm looking at the same code you
pointed at).

1) The code is always trying to read 4K at a time, and then *always*
writing 4K whether it's read 1 byte of 4K. I suggest you have a look at
http://www.pobox.com/~skeet/csharp/readbinary.html

2) It should also have a "using" statement around the FileStream (and
preferrably the MemoryStream too, for consistency) to make sure it
always gets disposed of when it's finished with, whether or not there
was an exception. (This is a fault in other methods too.)

3) Not an error, so much as bad style: It's using the
if (constant==variable) style which is a hangover from the bad-old days
of C, where if you wrote if (variable=constant) you wouldn't get a
compiler error. The idiom leads to more reliable coding in C, but at
the expense of a bit of readability.

4) In SendRequest(string uri, byte[] postBody, long postLength),
there's a lot of pointless looping around, writing 4K at a time, with
no indication as to why he doesn't just write the whole lot in one go.


Now, as to why you're running out of memory... I suspect the problem is
with how the MemoryStream is growing. You could try creating the
MemoryStream with enough space for the initial piece of text and
everything you need from the post body's data (using Stream.Length).
That would avoid MemoryStream having to copy everything each time it
resizes.

However, a better solution would be to refactor the code to avoid
building up the MemoryStream in the first place. There's no reason why
there shouldn't be a version of SendRequest which takes a stream to
write stream to the request object. Now, whether that actually streams
it to the client or whether it buffers it all up in memory anyway is
another matter, but it would be saving at least one 300M+ buffer!
 
W

Willy Denoyette [MVP]

|
| <snip>
|
| > Also keep in mind that strings in .NET are really System.Char arrays,
where
| > a char is 16 bit. So if your doc is ASCII encoded, it will take twice
the
| > size of the file in memory.
|
| Where is the file being treated as a string? The ASCII encoding is only
| used for the URL, as far as I can see.
|
|
Not in this piece of code, but I guess that's not all there is running. It
was just a warning, some may forget that when they threat the stream a
strings they take twice the size.

Willy.
 
T

TerryFei

Hi Simeon,

I just wanted to check how things are going. If there is any question,
please feel free to join the community and we are here to support you at
your convenience. Thanks for your understanding!

Best Regards,

Terry Fei[MSFT]
Microsoft Community Support
Get Secure! www.microsoft.com/security
(This posting is provided "AS IS", with no warranties, and confers no
rights.)
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Top