LINQ question - reading structured files

[Griff]

Hi

Is it possible to query a structured file that is a FIXED WIDTH file rather
than a delimited file?

I think it would be great to first define how many columns one is expecting
and the width (in characters) of each column, and then to be able to write
WHERE clauses in the Linq query e.g. "where field(2) = [mySearchTerm]".

But I can't find any examples on the web of how to achieve this...

Thanks

Griff

 

[Chris Dunaway]

Hi

Is it possible to query a structured file that is a FIXED WIDTH file rather
than a delimited file?

I think it would be great to first define how many columns one is expecting
and the width (in characters) of each column, and then to be able to write
WHERE clauses in the Linq query e.g. "where field(2) = [mySearchTerm]".

But I can't find any examples on the web of how to achieve this...

Thanks

Griff

I'm not 100% sure this would work (check my syntax), but you could
create an extension method which would parse the line into it's
appropriate parts and return a List(Of T) then query that using the
let keyword:

from line in File.ReadAllLines("filename.ext")
let parts = line.SomeExtensionMethod()
where parts(2) = "Some Search Term"
select new {Name=parts(0), City=parts(1), Title=parts(2)}

I'm not sure of this syntax, but the let keyword may allow you to
store the result of a method and query that further. I typed this
from my head so it may not work.

Chris

 

[Jon Skeet [C# MVP]]

I'm not 100% sure this would work (check my syntax), but you could
create an extension method which would parse the line into it's
appropriate parts and return a List(Of T) then query that using the
let keyword:

from line in File.ReadAllLines("filename.ext")
let parts = line.SomeExtensionMethod()
where parts(2) = "Some Search Term"
select new {Name=parts(0), City=parts(1), Title=parts(2)}

I'm not sure of this syntax, but the let keyword may allow you to
store the result of a method and query that further. I typed this
from my head so it may not work.

Note that in order to avoid reading the whole file in as text in one
go, you could use the LineReader class which is in my MiscUtil library:
http://pobox.com/~skeet/miscutil

For example:

var query = from file in Directory.GetFiles(@"c:\logs", "*.log")
from line in new LineReader(file)
let entry = new LogEntry(line)
where entry.Severity = Severity.Critical
select entry;

This will read a line at a time, which is probably a better idea when
the files are huge...

Note the other difference between the quoted code and mine - I haven't
created an extension method on string, I've just called a constructor.
I think it's worth avoiding creating *too* many extension methods,
particularly on common types such as string.

 

[Chris Dunaway]

Hi

Is it possible to query a structured file that is a FIXED WIDTH file rather
than a delimited file?

I think it would be great to first define how many columns one is expecting
and the width (in characters) of each column, and then to be able to write
WHERE clauses in the Linq query e.g. "where field(2) = [mySearchTerm]".

But I can't find any examples on the web of how to achieve this...

Griff

I'm not 100% sure this would work (check my syntax), but you could
create an extension method which would parse the line into it's
appropriate parts and return a List(Of T) then query that using the
let keyword:

from line in File.ReadAllLines("filename.ext")
let parts = line.SomeExtensionMethod()
where parts(2) = "Some Search Term"
select new {Name=parts(0), City=parts(1), Title=parts(2)}

I'm not sure of this syntax, but the let keyword may allow you to
store the result of a method and query that further. I typed this
from my head so it may not work.

Chris

My last post used c# syntax. You might also try something like:

From line In File.ReadAllLines("filename.ext")
Let parts = line.SomeExtensionMethod()
From part In parts
Select new Name=parts(0), City=parts(1), Title=parts(2)
Take While part.Title = "Search Term"


Chris

 

[Chris Dunaway]

Note that in order to avoid reading the whole file in as text in one
go, you could use the LineReader class which is in my MiscUtil library:http://pobox.com/~skeet/miscutil

For example:

var query = from file in Directory.GetFiles(@"c:\logs", "*.log")
from line in new LineReader(file)
let entry = new LogEntry(line)
where entry.Severity = Severity.Critical
select entry;

This will read a line at a time, which is probably a better idea when
the files are huge...

Note the other difference between the quoted code and mine - I haven't
created an extension method on string, I've just called a constructor.
I think it's worth avoiding creating *too* many extension methods,
particularly on common types such as string.

Thanks Jon, I wasn't sure if my code would even work since I pulled
from the air,. I don't have a machine available on which to test
it.

By the way, tell them to hurry your book along! When I pre-ordered,
it was supposed to be out in March (according to Amazon) but now I see
it's been pushed back to April!! Oh well, I'll have time to finish
"Linq in Action" by then, so maybe it's not too bad!

Chris

 

[Jon Skeet [C# MVP]]

Thanks Jon, I wasn't sure if my code would even work since I pulled
from the air,. I don't have a machine available on which to test
it.

Absolutely - it's always fun doing this kind of thing in an ad hoc
manner.

By the way, tell them to hurry your book along! When I pre-ordered,
it was supposed to be out in March (according to Amazon) but now I see
it's been pushed back to April!! Oh well, I'll have time to finish
"Linq in Action" by then, so maybe it's not too bad!

Don't worry, it's coming along nicely... I'm about to proof read
chapter 7 (as in, it's sitting next to me, waiting for me to finish
this post). The e-book should be available in March, with the real
paper copy in April.