Entity Framework on large model

[corentin Vimont]

Hello all,
We are trying to migrate to Linq to Entity for managing access to an SQL
Database. We are experiencing problems when dealing with an EDMX model
containing a lot of entities and relations between entities. Everything
works fine when we add 50 entities to the model, with an average of 2
Foreign keys per entity, but when adding 700 entities, the EDMX seems to
be overwhelmed and refuses to continue.

We have already tried the method consisting of splitting the model into
several thematic models, but are not satisfied with this, as we are
working on an ERP project, almost every table will have to be accessible.

We have already tried removing all the relations between the entities in
the model in order to lighten the requested memory, but Linq looses
most of it interest without FK management…

This point could force us to adopt another technology for accessing
databases if we don’t find other ideas. Does someone know a better way
of managing large model with several hundreds of entities without being
limited as we are now? We are surely not the first developer team using
Linq to Entity on a large database?

Thanks in advance for your answers.

Corentin Vimont

 

[sqlguru]

Linq, entity model framework, MVC framework, OR/M are all pathetic
hacks that does not belong in an enterprise environment. Only idiots
decide to use those 1-2-3-poof magic application RAD guaranteed to
fail techniques.

Get yourself a beginners level book on software engineering and design
patterns.

 

[Matthieu MEZIL]

ORM and particularly EF, is really cool because we can gain a lot of
productivity through it. Moreover, you can perfectly build great
architectures (not RAD) with it. I am not writing that it's the best way in
every case but I think that only idiots systematically reject it.
You probably never use Entity Framework. I wrote a training course about it.
Do you want me to put you in touch with one of my company salesmen to allow
you to keep up with it?

 

[Rochdi]

Dear Sql Guru, I do really wonder if you’re able to think out of SQL. And I
think that you’re showing the symptoms of a deep impedance mismatch between
your SQL world and the dangerous object oriented world. Linq, entity model
framework, MVC framework are not meant to enterprise applications?? !!! You
need really an intensive cure

 

[sqlguru]

I'm actually more of an applications developer than a DBA.
Tell me something, can you lazy-load in your entity framework? NO.
Can you implement complex business logic (with multiple dependent
transactions, persistence validations, complex "dynamic" search
functionality)? NO.
Can you implement a multi-tier cache? NO.
What about PROPER source control? NO.
Can you use it for very large projects? NO. Have you tried to use
Visual Studio when you have over 1000 entities and you are using the
entity framework? You will need a supercomputer!

MVC framework is a joke. Only idiots use the entity framework.

Get yourself a book on database and application architecture and so
you can stop writing 1-2-3-poof magic applications that display
"Server timeout" and "Application not responding".

 

[Daniel Simmons]

The entity framework is not designed for models with large numbers of
entities. Not only does it cause perf issues to load the entire large model
at once, but it is a key recommended design practice to divide your domain
into smaller bounded contexts. This is important for a variety of reasons.

First off, it makes things more understandable and manageable for
programmers working in a particular module. If you did have a single model
with 700 entities, then what would the experience be like when you type the
name of your context in the VS editor and then hit "."? Intellisense becomes
completely worthless. It's important to divide your application into
reasonable chunks that you can work with in a particular situation.

Secondly, we often discover that the real meaning and business logic for a
particular entity is slightly different in different modules. So it's very
useful to have different, module-specific "views" of entities that are shared
across modules.

And of course performance not just of the EF but of any framework is
affected if you try to reference too many things in one place. (How does
DataSet do if you load all 700 tables into a single DataSet?)

Usually we find that large databases partition into fairly separable sets of
entities possibly with some small number of things shared across the modules.
You can read some about this approach in a couple of blog posts on the
ado.net team blog:

http://blogs.msdn.com/adonet/archiv...-large-models-in-entity-framework-part-1.aspx
http://blogs.msdn.com/adonet/archiv...-large-models-in-entity-framework-part-2.aspx

You might also want to take a look at this post from Matthieu Mezil on the
subject:

http://msmvps.com/blogs/matthieu/archive/2009/05/27/how-to-split-your-edm.aspx

- Danny

 

[sqlguru]

Also, take a look at the readability of that crap (the linq to object
query). I guarantee the SQL solution can be easily understood and
requires fewer lines than that "hack". .NET is trying to be a All-In-
One-Printer.

 

[Miha Markic]

MVC?

 

[Tony]

I have been developing ERP solutions using object model based development on
IBM's iSeries (AS/400) platform for 19 years. My current Customer has about
900 entities. The model loads in 5 seconds. Our developers have achieved
many fold of productivity improvement and the code is rock solid.

The challenge I have attempting to move to or utilize the Microsoft Entity
Framework is that it is not scalable beyond a simple classroom model. It is
a good start. However, it’s not ready to develop serious large business
applications.

In fact, before Microsoft continues to develop the framework, the
fundamental foundation architecture needs to be redesigned. The architecture
is not scalable beyond a handful of related entities.

It is not scalable because the model is managed as one large chunk of bits
rather than a logical grouping of a bunch of small sets of bits. This all or
nothing approach is not scalable as computers have physical limitations.

The run time is just as bad as development. The pre-compiling helps but
it’s an afterthought and clunky to use or remember to use. Why does all this
stuff need to be preloaded as one huge chunk? Why do I have to remember to
pre-compile? Why not produce high-performance code out of the chute? Why
not load as little as needed only when it is needed. Dynamic is good but why
make it so dynamic that the overhead of managing the framework leaves little
resources left to run the application.

It is not scalable because the diagramer requires that ALL related entities
to appear in the diagram. One diagram works great in the classroom but fails
in real life. An entity should stand on its own and be able to appear in
zero or more diagrams. (Talk the SQL Server diagram folks. They do it.) A
separate diagram can then represent each subject such as Customer,
Purchasing, Inventory, etc. The diagram does not have to be the same thing
as the model, they can be two different things.

Today, breaking the model into several entity diagrams can only work well if
you have a physical separation in relation between the objects. This is not
the case in a large ERP system. Even if the entity diagram was broken into
several diagrams, it still wants to process the entire model, although it is
smaller, as one huge chunk.
The foreign key and relationship functionality of the Entity Framework and
LINQ looks intriguing and I can’t wait to use it. However, its meta data
needs to be re-architected to be scalable to thousands of entities with each
able to be represented in multiple diagrams. Although it works fine in the
classroom, one big blob of bits doesn’t work in real life.

-- Tony