HELP!! FAST way to read Parts of Big Files

[DraguVaso]

Hi,

I have files I need to read, which contains records with a variable lenght.
What I need to do is Copy a Part of such a File to a new File, based on the
a Begin- and End-record.

I used this functions:
Dim intMyFile As Integer = FreeFile()
FileOpen(intMyFile, MakePathFile(strDirS, strFileS), OpenMode.Input,
OpenAccess.Read, OpenShare.Shared, -1)
Do While Not EOF(intMyFile)
strLine = LineInput(intMyFile)
If (intX >= intStartRec) And (intX <= intEndRec) Then
strNew = strNew & strLine & vbCrLf
End If
intX = intX + 1
Loop

It worked fine until I met some really big files. I have some files of 10
Mb, containing 75000 records... After 20 minutes my application still
doesn't have read the exact part.


I tryed this:
Dim fsFile As New FileStream(MakePathFile(strDirS, strFileS), FileMode.Open,
FileAccess.Read)
Dim brFile As New StreamReader(fsFile,
System.Text.Encoding.GetEncoding(1252), False, fsFile.Length - 1)
intX = 0
Do While intX <= intEndRec
strLine = brFile.ReadLine
If (intX >= intStartRec) And (intX <= intEndRec) Then
strNew = strNew & strLine & vbCrLf
End If
intX = intX + 1
Loop

But it's as slow as the other one.



Only one thing was really quick (only 10 seconds):
Dim fsFile As New FileStream(MakePathFile(strDirS, strFileS),
FileMode.Open, FileAccess.Read)
Dim brFile As New StreamReader(fsFile,
System.Text.Encoding.GetEncoding(1252), False, fsFile.Length - 1)
Dim strChar((intEndRec * 128) - 1) As Char
For intX = 0 To strChar.Length - 1
strChar(intX) = " "
Next
brFile.ReadBlock(strChar, intStartRec * 128, (intEndRec - intStartRec) *
128)

But here I had several stupid problems for which I din't really find a
solution:
- first of all: I'm having really big problems converting the Char to a
String (I tryed filing the Char with spaces and than Trim it but what about
spaces in the end of my File?)
- the ReadBlock works with character-positioning, and not with lines. Is
there a way to convert a line-position to a character-position or do a
ReadBlock with lines or something like that?


Anyhelp would really be appreciated!! I'm really stuck with this problem,
and it's kidn of urgent!! Any help regarding the ReadBlock or
StremReader-stuff, or on other (FAST!) way to do this would really be
appreciated!!

Thanks a lot in advance!!

Pieter

 

[Jon Skeet [C# MVP]]

I have files I need to read, which contains records with a variable lenght.
What I need to do is Copy a Part of such a File to a new File, based on the
a Begin- and End-record.

I used this functions:
Dim intMyFile As Integer = FreeFile()
FileOpen(intMyFile, MakePathFile(strDirS, strFileS), OpenMode.Input,
OpenAccess.Read, OpenShare.Shared, -1)
Do While Not EOF(intMyFile)
strLine = LineInput(intMyFile)
If (intX >= intStartRec) And (intX <= intEndRec) Then
strNew = strNew & strLine & vbCrLf
End If
intX = intX + 1
Loop

It worked fine until I met some really big files. I have some files of 10
Mb, containing 75000 records... After 20 minutes my application still
doesn't have read the exact part.

The problem is your string concatenation. Either create a StreamWriter
to start with, and just copy the relevant lines straight to disk, or
use a StringBuilder to build up the string in memory.

Just reading through the file should be very fast indeed.

 

[DraguVaso]

Shit :-S
*enormous blush* :-(
I spend 2 hours trying to get it faster and getting frustrated, and I didn't
think about the string-concatenation :-(

Thanks a lot! You made my day! hehe

Pieter