Linq Over Dataset/SQL Server in terms of Performance, Concurrency

[Guest]

Hi Experts,

I am doing a prototype of providing data access (read, write & search)
through Web Service. We observed that the data storing in SQL Server 2005,
the memory size is always within 250MB. Our aim is to support ~50K
concurrency users.



After investigation, we are thinking to use In-memory database for achieving
higher goal. One of choices to use LINQ as data structure while I would like
to ask some considerations in the meanwhile.

- How is the performance, Concurrency Issues & Memory Management using LINQ
over Dataset and SQL server ?

Many Thanks.

 

[Jon Skeet [C# MVP]]

I am doing a prototype of providing data access (read, write & search)
through Web Service. We observed that the data storing in SQL Server 2005,
the memory size is always within 250MB. Our aim is to support ~50K
concurrency users.

After investigation, we are thinking to use In-memory database for achieving
higher goal. One of choices to use LINQ as data structure while I would like
to ask some considerations in the meanwhile.

- How is the performance, Concurrency Issues & Memory Management using LINQ
over Dataset and SQL server ?

The memory requirements will very much depend on the data. I've
recently been working on a project with a 7GB database which fits into
~100MB in memory due to a few bits of cunning. Other cases might be
worse than the database.

What I'd be worrying about is the writing and searching side of
things. Would you write to memory and also write back to the database
at the same time? Do you have multiple servers, and would they have to
synchronize? (50K concurrent users sounds like a lot for one server,
but it depends on what you really mean by concurrent users.)

In terms of concurrency, Joe Duffy is working on a "PLINQ" project to
utilitise concurrency effectively within LINQ, but I don't know many
details about it.

Sorry not to have more hard facts to give you, but it really depends
immensely on what your data is like and exactly what you need to do
with it.

Jon

 

[Laura T.]

Hard to say much without knowing the underying data and it's organization.
For 50k and more users (concurrent=resource consuming?), the overall
architecture becomes quite critical.

Some, hopefully, useful pointers:

http://blogs.msdn.com/wriju/archive/2007/06/08/linq-performance.aspx
http://www.davepeck.org/oldessay/linq-collection-perf.html
http://www.bluebytesoftware.com/blog/PermaLink,guid,200c3151-fbd5-4bfe-bb1e-0d6b90c6442b.aspx

 

[Guest]

Thanks experts, let me tell more about the story.

Scenarios:
There are a few (one or two) tables storing some information of servers (ip,
name, other fields).
The system allows other users add/remove/search by keywords for the table(s).
It is estimated that there would be 50K concurrency users.
The content of table(s) is changing while the total size of table is always
within 250MB (when we store it in SQL Server).

Background:
Our previous project is to use SQL Server05 while the bottomneck is at the
ORM stuff in connecting to SQL Server. So, I would like to investigate if it
is in the case of in-memory db.

And I would like to ask about performance, Concurrency Issues & Memory
Management of LINQ over SQL/DataSet.
Many Thanks.

 

[Nicholas Paldino [.NET/C# MVP]]

I don't think that Linq for DataSets is going to help you here, due to
what I believe (based on what you said) the frequently changing content of
the tables. If you load the database into memory, you still will have to
persist the changes to the dataset in memory back to the database, and at
the least, you are going to have the same overhead on update operations that
you would have if you were just using the database.

On the select side, you might get a performance increase, but honestly,
with that many concurrent users, you will have to manage the concurrency to
updating and selecting that data which to me seems like a database is much
better suited for.

I think that you should take a look at how you are querying the database
and see what optimizations you can make there. You would be surprized at
the kinds of optimizations you can make using indexes, stored procedures,
changing the logic in queries, etc, etc.