N
nlbauers
Hi all,
I am looking for suggestions for replacing a very large 2D double
array in my application with some kind of out of RAM storage. The
data stored in the array will soon be growing so large that it will
impact application performance by sucking up most of the system RAM.
We expect the existing design will soon require an array of over 800MB
of space and future requirements will need even more. The big issue
we have is access time- the solution should minimize impact on the
application execution. Currently the array is written to once at
application startup, it is then read on a single element basis
(meaning elements are picked out here and there, not looping over
large portions of the array) fairly often during normal application
execution, and the values are updated infrequently. That means that
single element access is the thing we are the most concerned about
dragging down application performance. So, here are the current
suggestions:
1) Replace the in-memory array with a database. The plus side here is
that in-memory size wont be an issue. The down side here, is that
accessing single records/elements (a fairly common operation) would be
an order or magnitude more expensive. My testing indicates 500x
slower to access a single database record.
2) Replace the in-memory array with a binary file. My testing
indicates that single element file access would be slightly faster
than a database query, but still an order of magnitude slower than in-
memory array access.
3) A buffered hybrid storing the most commonly accessed elements in-
memory and paying the performance penalty when retrieving those that
are not used as often. This is the solution we are leaning toward.
Anyone else have a clever suggestion for solving this problem? Seems
to me this problem must have been solved before.
The solution will be implemented in C# or C++ if it matters.
Neil
I am looking for suggestions for replacing a very large 2D double
array in my application with some kind of out of RAM storage. The
data stored in the array will soon be growing so large that it will
impact application performance by sucking up most of the system RAM.
We expect the existing design will soon require an array of over 800MB
of space and future requirements will need even more. The big issue
we have is access time- the solution should minimize impact on the
application execution. Currently the array is written to once at
application startup, it is then read on a single element basis
(meaning elements are picked out here and there, not looping over
large portions of the array) fairly often during normal application
execution, and the values are updated infrequently. That means that
single element access is the thing we are the most concerned about
dragging down application performance. So, here are the current
suggestions:
1) Replace the in-memory array with a database. The plus side here is
that in-memory size wont be an issue. The down side here, is that
accessing single records/elements (a fairly common operation) would be
an order or magnitude more expensive. My testing indicates 500x
slower to access a single database record.
2) Replace the in-memory array with a binary file. My testing
indicates that single element file access would be slightly faster
than a database query, but still an order of magnitude slower than in-
memory array access.
3) A buffered hybrid storing the most commonly accessed elements in-
memory and paying the performance penalty when retrieving those that
are not used as often. This is the solution we are leaning toward.
Anyone else have a clever suggestion for solving this problem? Seems
to me this problem must have been solved before.
The solution will be implemented in C# or C++ if it matters.
Neil