DataSet Memory Consumption

[TK]

Hello,

I'm trying to load about 600MB of mysql data into a
DataSet which seems to be increasing the size of the data
in memory by a factor of 3.5 times to 2.1 GB.

The server has 4GB of RAM and i want to cache the db data
at application_start to avoid any further database access.
i'm willing to give up about 1GB but not 2.1 GB.

Is there a way to make a DataSet use less memory?
Thanks,
TK

 

[Bernie Yaeger]

Hi TK,

Caching a database simply doesn't make sense. I know this is not your
question, but what do you hope to gain by doing it? Loading tables into
datasets/datatables is simple enough, even when their quite large. Often
you can 'narrow' the cols returned and surely filter the rows returned to
make it very simple.

HTH,

Bernie Yaeger

 

[William Ryan]

TK, that's trouble. Grab what you need as needed. You can technically grab
that much data, but it's doubtful that 600mb will be needed all at once.
Think of streams as an analogy. If you had to push over an entire file each
time you wanted to play audio from a server, you'd be very limited in what
you could play, or, you'd need some pretty serious machinery. If your
requirements are such that you actually need all 600MB, it's not going to be
cheap, more ram and better processors are probably the only solution.

 

[Guest]

Hello Bernie Yaeger,

thanks for the response.

you don't believe there will be a difference in speed if i
access data from memory rather than hard drive? but then
again databases are maybe optimized enough to minimize the
difference in speed.
Thanks,
TK

 

[Bernie Yaeger]

Hi TK,

The difference in speed on my systems has been negligible. That is, while I
never tried to cache a database, my apps open hundreds of tables and joins
in seconds. As I mentioned, only when trying to load a 25 column 150,000
row table does it require some time - say 20 - 25 seconds. When it gets
really large - 650,000 rows - it takes too long. But in almost every case I
give my users ways around it - invoice history starting at a select date -
and very often meeting specific criteria - acct # = etc - and narrowing of
cols - select acctnum, inv_dt, amt from invdetails etc.

HTH,

Bernie