problem with System.OutOfMemoryException

[Lloyd Dupont]

In my code I am manipulating/creating/destroying in memory representation of
big XML file (as in a few MB of CDATA in them (embeded bitmaps)).

At some stage (and I am in no way under any memory stress as my application
is using only about 500MB of memory and I have 1GB free, without counting
the virtual memory) I have this out of memory exception happening, in code
snippet looking like that:
=====
public static void WriteXml(System.Xml.XmlWriter writer,
IEnumerable<string> resources, IResourceProvider provider)
{
writer.WriteStartElement("Resources");

writer.WriteElementString("ResourceProvider", provider.ID.ToString());
List<string> resourcesWritten = new List<string>();
foreach (string resource in resources)
{
if (resourcesWritten.Contains(resource))
continue;
writer.WriteStartElement("Resource");
byte[] data = provider.LoadData(resource);
writer.WriteElementString("ResourceName", resource);
string b64Data = Convert.ToBase64String(data);
// ========== OutOfMemoryException below
writer.WriteElementString("base64Data", b64Data);
writer.WriteEndElement();
resourcesWritten.Add(resource);
}
writer.WriteEndElement();
}
=====
the error stack is:
====
System.Xml.dll!System.Xml.XmlWellFormedWriter.WriteString(string text) +
0x55 bytes
System.Xml.dll!System.Xml.XmlWriter.WriteElementString(string localName,
string ns, string value) + 0x29 bytes
System.Xml.dll!System.Xml.XmlWriter.WriteElementString(string localName,
string value) + 0xb bytes

NovaMind.Data3.dll!NovaMind.Data.ResourcesHelper.WriteXml(System.Xml.XmlWriter
writer = {System.Xml.XmlWellFormedWriter},
System.Collections.Generic.IEnumerable<string> resources =
{NovaMind.Data.NMMapBranch.get_ResourcesInUse},
NovaMind.Data.IResourceProvider provider =
{NovaMind.Data.NMDocumentPackage}) Line 27 + 0x13 bytes C#

====
When I look at the internal of the various object involved with the debugger
I could see that:
data.Length ~= 7 Mb (~ 7 e+6)

and "Writer" is wrapper around a StringBuilder which MaxCapacity is around 2
GB (2e+9) and which current Length is "only" about 25 Mb (25e+6).

Now, all these numbers are huge I agree, however my memory isn't stressed at
all (i still used much less than my physical memory, virtual memory
notwithstanding)).
Any idea why I have OutOfMemory exceptions? how to get rid of them?

 

[Niels Ull]

My guess would be that your memory is too fragmented to provide a continuous
25 Mb block of memory for the string.
How about using the built in WriteBase64 method instead? It probably doesn't
build a string with the entire base64 string.
E.g.

writer.WriteStartElement("base64Data");
writer.WriteBase64(data, 0, data.Length);
writer.WriteEndElement("base64Data");

Or, if that fails, do it in pieces, e.g.

writer.WriteStartElement("base64Data");
for(int i = 0; i < data.Length; i += 1024)
writer.WriteBase64(data, i, Math.Max(1024, data.Length-i));
writer.WriteEndElement("base64Data");

Regards

Niels