Best Practices with In Memory Data

[Ben Kim]

Hello all,

I am coming from another development environment (Clarion for Windows) and
they offer a special In-Memory driver which acts just like a database
driver. What would be the appropriate replacement for such an animal using
VB.NET 2005? We are looking for the fastest possible speed, data does not
have to persist (may in some cases) and could be globally scoped or local to
the procedure. Would you choose a collection or an array to handle this
type of functionality? Can I bind this collection/array to a data grid box?

Thanks in advance!

Ben Kim
Emergitech

 

[Cor Ligthert [MVP]]

I am coming from another development environment (Clarion for Windows) and
they offer a special In-Memory driver which acts just like a database
driver. What would be the appropriate replacement for such an animal
using VB.NET 2005? We are looking for the fastest possible speed, data
does not have to persist (may in some cases) and could be globally scoped
or local to the procedure. Would you choose a collection or an array to
handle this type of functionality? Can I bind this collection/array to a
data grid box?

Dataset/datatable although it is not a real SQL based in memory database has
it all functionality to to the same.

 

[Ben Kim]

Excellent - Thanks Cor just what I needed to know!

Ben Kim

 

[CMM]

I would agree with Cor. Datasets are excellent in-memory "databases."

On the other hand, arrays and the myriad of collections and dictionaries in
the System.Collections namespace are cool too... and may be more
efficient... but in terms of performance and robustness datasets can't be
beat.

To answer another one of your questions: yes you can bind both arrays and
collections to controls. They all implement the IList interface. BTW,
collections are really not much more than wrappers around an array. Again,
the dataset is the Zeus of all "lists."

 

[Jim Wooley]

There are a number of options out there. I'm not sure that I agree with the
recommendation on passing around datasets. They do have a fair amount of
bloat which you may not want if you require "fastest possible speed". There
are a number of factors to consider, including physical distance between
entities, network connection speeds, targeted platforms, etc.

Your fastest option is binary formatted remoting over TCP/IP. However, this
will only work between .Net applications and may require firewall
modifications to allow clients to access your service.

The more accepted method (SOAP) packages your data as XML inside of a
envelope (more XML) and transmits over HTTP (typically using IIS). This is a
more open standard as any platform can create and consume XML. It is human
readable as well which has it's advantages (easier to read thus easier to
program) and disadvantages (easier to read, thus harder to secure).

In between these options, there are a myrad of possibilities. I strongly
suggest you start looking at WCF (formerly code named Indigo) as it allows
switching between some of these options using command line switches. If you
are just starting a project now, you may be able to time your product's
release along with VISTA/WinFx (by all accounts before the end of the year).

I recommend you run some feasibility studies comparing the performance of
datasets compared with collections of objects. Both can be serialized using
XML. The objects can be more customized for your particular business needs
and do not incur the overhead of additional functionality you may or may not
need. As always, YMMV.

Jim Wooley

 

[CMM]

I agree with you. Like you said, the best option is "remoting."

But, you don't want to use datasets because of efficiency concerns... those
same concerns should preclude you from using SOAP. The only place for SOAP
is for interoperability (like you need to communicate with a Java
webservice.). SOAP is the absolute King of Bloat. Not only is the envelope
verbose XML but even if you decide to put a binary payload in it, it has to
be converted to a tranportable encoding which doubles even triples the size
of the payload. Putting serialized data as XML in it is even worse. I don't
hate SOAP or WebServices... I just think its retarted to use them in an
internal LAN... especially when regular ol' Remoting provides so many more
benefits.

 

[Cor Ligthert [MVP]]

Jim,

Datasets don't have to be serialized they are serialized.

As CMM write can it be a minor that it sends as well description
information. However it can be a plus if the information has to sent a lot
of empty fields. Those can be avoided to sent with a dataset.

Just my thought,

Cor

 

[Jim Wooley]

I think we are in agreement. I mentioned SOAP as he is needing to integrate
systems potentially. If they aren't all in .NET, passing the dataset may not
be an option, nor would remoting. I agree that the slowest option would be
SOAP over HTTP and the fastest is Remoting over TCP/IP. He will need to
weigh the business needs (interconnected systems) with the performance needs
(fastest possible) to determine the final architecture.

Jim Wooley