I'm really stumped on this one and would appreciate some
pointers...

I have an app that allows importing certain data in a
dataset (at which point its all in memory) and then use
BinaryFormatter to store the dataset and some more stuff
in a binary file as my application's data file. This all
works fine with smaller files, but some of the datasets
can be as large as 100 Meg when loaded in memory. The
import operation works fine, but when I try to serialize
my app's data in such a case using BinaryFormatter to
FileStream, I get a OutOfMemoryException. I have about
512M RAM. Only once was I able to write it successfully
without getting the OOM exception, and noticed that the
peak memory usage for the process went up to over a gig. I
have no idea why, and have tried a number of different
things such as explicitly specifying the target stream as
File in the context for the BinaryFormatter, but don't
seem to be able to avoid the exception.

Any thoughts on how I could handle this so that it can get
written out and read back from the file? I do want to
support 100M datasets in memory, so using SQL server, etc.
is not an option. This strictly has to be a file based
dataset.

Thanks in advance.

Jitu

Theres an issue with serialization of big datasets, internally I think it
creates a huge amount of small objects and eventually takes up all your
memory.
Even by using the BinaryFormatter your dataset will end up serialized in
xml. Instead use DataSet.WriteXml() methods to write and ReadXml() to read
from that file. There is no problem with memory using those methods.

Chris.

"Jitu" <(E-Mail Removed)> wrote in message
news:02b301c3d7ea$4e5fb4c0$(E-Mail Removed)...
> I'm really stumped on this one and would appreciate some
> pointers...
>
> I have an app that allows importing certain data in a
> dataset (at which point its all in memory) and then use
> BinaryFormatter to store the dataset and some more stuff
> in a binary file as my application's data file. This all
> works fine with smaller files, but some of the datasets
> can be as large as 100 Meg when loaded in memory. The
> import operation works fine, but when I try to serialize
> my app's data in such a case using BinaryFormatter to
> FileStream, I get a OutOfMemoryException. I have about
> 512M RAM. Only once was I able to write it successfully
> without getting the OOM exception, and noticed that the
> peak memory usage for the process went up to over a gig. I
> have no idea why, and have tried a number of different
> things such as explicitly specifying the target stream as
> File in the context for the BinaryFormatter, but don't
> seem to be able to avoid the exception.
>
> Any thoughts on how I could handle this so that it can get
> written out and read back from the file? I do want to
> support 100M datasets in memory, so using SQL server, etc.
> is not an option. This strictly has to be a file based
> dataset.
>
> Thanks in advance.
>
> Jitu
>