Linq or XmlReader

[CSharper]

I have a project where I need to read a decent size (5MB) files for
processing. I have two of them, one of them I need to iterate through
a collection (using Linq) and find associated elements in the other
XML file. Can I use Linq in both the situation (since coding is easy)
or use XMLReader for one and Linq for other? If I remember correctly,
for Linq to work XML has to be loaded in the memory.
Thanks,

 

[Martin Honnen]

I have a project where I need to read a decent size (5MB) files for
processing. I have two of them, one of them I need to iterate through
a collection (using Linq) and find associated elements in the other
XML file. Can I use Linq in both the situation (since coding is easy)
or use XMLReader for one and Linq for other? If I remember correctly,
for Linq to work XML has to be loaded in the memory.

Yes, LINQ builds an in memory tree model of your XML so its memory
consumption grows with the size of the XML. On the other hand 5MB does
not sound that large these days for parsing files in a Windows desktop
application.

You can also combine XmlReader and LINQ to XML using the XNode.ReadFrom
method:
http://msdn.microsoft.com/en-us/library/system.xml.linq.xnode.readfrom.aspx

 

[Martin Honnen]

I have a project where I need to read a decent size (5MB) files for
processing. I have two of them, one of them I need to iterate through
a collection (using Linq) and find associated elements in the other
XML file. Can I use Linq in both the situation (since coding is easy)
or use XMLReader for one and Linq for other? If I remember correctly,
for Linq to work XML has to be loaded in the memory.

Yes, LINQ builds an in memory tree model of your XML so its memory
consumption grows with the size of the XML. On the other hand 5MB does
not sound that large these days for parsing files in a Windows desktop
application.

You can also combine XmlReader and LINQ to XML using the XNode.ReadFrom
method:
http://msdn.microsoft.com/en-us/library/system.xml.linq.xnode.readfrom.aspx

 

[Paul]

Why not create a model structure to represent the xml data and create a DAL
using XMLReader. You can then carry out BLL on the model objects when
loaded. There is a good chance you can use the model structure elsewhere
too.

XML is a datasource as much as any database is, even if that XML is the
result of an HTTP POST or FTP.

Also when you are doing set operations you may find a dictionary far easier
to use and probably although I have never tested it far quicker than
constantly iterating though lists for matches. Even if you just create an
index of a list using a dictionary during population of the list.

You may also want to consider integration services on SQL Server. Once the
data is in a temp DB it can probably churn the data far more efficiently
than any code you could write.

Just some alternative options to Linq. Easy code but less performant by a
magnitude.

 

[Paul]

Why not create a model structure to represent the xml data and create a DAL
using XMLReader. You can then carry out BLL on the model objects when
loaded. There is a good chance you can use the model structure elsewhere
too.

XML is a datasource as much as any database is, even if that XML is the
result of an HTTP POST or FTP.

Also when you are doing set operations you may find a dictionary far easier
to use and probably although I have never tested it far quicker than
constantly iterating though lists for matches. Even if you just create an
index of a list using a dictionary during population of the list.

You may also want to consider integration services on SQL Server. Once the
data is in a temp DB it can probably churn the data far more efficiently
than any code you could write.

Just some alternative options to Linq. Easy code but less performant by a
magnitude.