Text Files and arrays

[MikeJ]

Hi all....
I need to work with large text files....i know there is a lot of ways under
net to work with them....
i want speed, High Percentage of correctness

so i was thinking ...but not sure the best way to handle text files for
parsing into reports, or into SqlDatabase
or Where Ever else

my question is;
I can build a class which i did under win32 low level file io that only
stores the line offsets in a Array, which is quit fast i was able to load a
million plus line file in less than 6 seconds , so most files that are only
in the megabytes were a blink of a eye. then the class wrapper moves
pointers (not the data on disk) i only accessed disk when data was required,
worked pretty good.

under net i can load the whole file into a single dim array
i can calculate the array dim prior to load so there are only 2 sizing , the
creation of the array and at the end of load, sizing down to what i actually
used...

which is the best way to handle the files in .NET
Just load the The Array/collection with the rows in the file
or
Just load the line offsets into a array and process rows as required

Tks MJ

 

[AlexS]

I would go with offsets.

Do you expect files greater than 1Gb?

 

[MikeJ]

yes i do expect files greater than 1g....not often but will happen from time
to time....
im finishing the class that does offsets...and my current
testiing...1.1million rows 147meg takes about 7
seconds to load the offsets, which include
end of line crlf pair, cronly only and mixture of crlf and lf markers
i just go throw the file once....
i create the long[] with a size larger than required
estimated linelength/6
subtract the remainder from estimated length devide that by filesize puts me
a few elements over and above elements needed to load the file and traverse
it once.
Still working on the math above to get the array size closer to what is
needed.
so there is only 2 Array sizing...1=Creation 2=resize to ending size
(actually loaded offsets).


thanks for you advice
MJ