Working with sorted fixed length text files ?

[Thomas]

Hello !

I must develop an application for a WindowsCE.NET PDA with the .NET
Compactframework.

Due to compatiblity reasons with the backoffice application, the PDA
will have to receive and send back sorted fixed length text files (1
record per line, sorted in use order). I must precise that, at least
for one of the files, the number of records can be 10 000 max.

My questions regarding this situation are :
- would it be interesting to transform these files in XML files and
then to use Dataset / Datatable to manipulate it ?
- would it be interesting to transform these files in XML files and
then to use directly XML ? In this case, what about the methods to
search data and the performances ?
- or, last solution, since the files are sorted, could I use directly
text files ? At the moment, I think it would be the best solution but I
don't know the best way to search in such a file. If someone could give
me some clues or samples, it would be great.

I'm waiting for your advices, guys !

Regards,

Thomas.

 

[Peter Foot [MVP]]

If the files you are dealing with are large, and already in the right order
for you to process it wouldn't be that beneficial to read the whole file
into memory into XML or DataSet for example. However it can be done, take a
look at the OpenNETCF.Data.Text.TextDataAdapter (www.opennetcf.org/sdf/) for
example
Since your data is in lines you can create a StreamReader with your
filename, then call ReadLine to pull out each row. Then you can split this
using your delimiting character e.g. comma, tab etc to get your array of
fields. You may then parse these to specific types. Once you have then done
whatever processing you require on that row you could concatenate your
values back into a string for each row and write them to file using a
StreamWriter, then repeat the process for each row.
As I said previously I'd avoid reading in the whole file if possible,
however if you go through the source code for the TextDataAdapter you should
find parts of it are useful. Source code is in the vault
http://vault.netcf.tv/VaultService/VaultWeb/login.aspx (username: guest,
password guest)

Peter

 

[Thomas]

Thank you Peter for your quick answer.
If I decide to use text files, what is the best/speedest method to go
to a specific line/record ? Do I have to create a StreamReader and then
to read each line to find the first good one ?

Thank you.

 

[Guest]

If it's already sorted, and fixed length it's easy. Use a StreamReader and
a binary search to find the record you're after. With 10k lines it will be
a maximum of 13 reads to find any desired record.

-Chris

 

[Thomas]

Thanks again. Could you give some links or some explanations about the
way to implement a binary search on a text file ?
Please apology but I've never worked with this type of files...

Regards,

Thomas.

 

[Paul G. Tobey [eMVP]]

Just like a binary search on anything else. Figure out how many records
there are by looking at the file size and the length of the record. You
might wrap the retrieval of record #x so that you don't have to deal with
the record size calculation all over the place. Set your pivot variable to
1/2 way through the total number of records and read that record. If the
value you're searching for is less than the current record, move the pivot
to 1/2 way between the first record and the current pivot and set the last
record to the current pivot -1. If the value is greater than the current
pivot, set the first record to the current pivot +1 and set the new pivot to
1/2 way between that and the end record. Repeat until the end record number
is less than the first record number or until you've found the record you're
searching for.

Paul T.

 

[Thomas]

OK, Thank you everybody for your precise answers. I will try to use
binary search maybe combined with datatable when I will have to read a
set of records.

Regards,

Thomas.