ReadXML OutOfMemoryException

[joey.powell]

Hello guys, I have a pretty simple app that reads XML data (.ReadXML)
into an in-memory dataset, runs some linq queries, mods the data, and
then writes the XML (.WriteXML) back to disk.

Everything works well until, occasionally, I get an
OutOfMemoryException. So far this has only happened for a file that
was about 50MB in size. All other, smaller files are working just
fine.

50MB (+overhead of other objects etc...) certainly does not seem like
an unreasonable amount of data to hold in memory. How can I get this
to work without the error? Can I manually allocate more memory for the
DataSet and .ReadXML? Why doesn't it take advantage of more of the
available memory?

I know that there are lots of other ways to do this...but I am not
interested in reengineering the code. I just want it to work, because
I have since moved on to other projects. Is there a "quick and dirty"
way to make my app able to handle the 50MB of data in memory? This
does not seem like it should be such a big deal.

Apparently the .ReadXML and .WriteXML methods of the DataSet are not
very scalable!?

 

[cfps.Christian]

I'm actually fighting a bug that has something to do with this. I was
told that a few years ago Microsoft "fixed" something that made it so
their file transfer buffer is somewhere around 50MB. What this means
to a developer is there is a chance for a System exception if you try
to move more than that.

I haven't tested my code yet but right now I've got it altered so it
only moves 50MB at a time to hopefully get around this problem.

 

[Moe Sisko]

In the past, I've resorted to reading the big file a line at a time, and
breaking it down into smaller sized temp files. A low tech solution, and
you may need to be careful when splitting up the file depending on what
assumptions you can make about the format of the big file.

Moe