Text Stream Prefetch Question

[Guest]

Hi all,

I hope I am posting to the right MSDN group. I have some code that
loads a text file line by line. It works great. The issue I seem to see is
that it stalls every 500-600 lines and has to wait a second or so. I think
this is related to Access waiting on the system to load the file from disk.
My processing code is pretty effecient and can process lines really quick. I
am wondering if there is a way to look at how much data the OS prefetches
into memory for processing and if there is a way to increase it.

My code to open the file is below.

Sub XYZ (multiple parameters that initialize variables including filename)
Dim DevN As Integer
Dim SSQL, DBase, Tble, FS, CurFile As Variant
Dim CurLine, Temp, Unit, LOF, ProcessLine As Variant
Dim TempLine, ScrapA() As Variant

Set DBase = CurrentDb

Set FS = CreateObject("Scripting.FileSystemObject")
Set CurFile = FS.OpenTextFile(FileN, ForReading, TristateFalse)

While Not (CurFile.AtEndOfStream)
CurLine = CurFile.ReadLine
Wend

LOF = CurFile.Line

CurFile.Close

Set Tble = DBase.OpenRecordset("All Data")

Set CurFile = FS.OpenTextFile(FileN, ForReading, TristateFalse)

CurLine = ""

While UBound(Split(CurLine, ",")) <> 9 ' 10 items in the Desired Header
CurLine = CurFile.ReadLine
Wend

(Some limitied pre-processing and setup done here, but no file processing)

CurLine = CurFile.ReadLine

While Not CurFile.AtEndOfStream
(line processing occurs here)
CurLine = CurFile.ReadLine
Wend

CurFile.Close

End Sub

Thanks in advace for the help.

Regards,
Texernie1

 

[Stephen Lebans]

Depending on the size of the Text file read the entire file at once into an
array and then process the array.

--

HTH
Stephen Lebans
http://www.lebans.com
Access Code, Tips and Tricks
Please respond only to the newsgroups so everyone can benefit.

 

[Guest]

Stephen,

Thanks. That might work.

Is there a limitation on how big an array can be? It would be feasible
to have array sizes of 500,000 elements or more and common 100,000 or more.
The file sizes are anywhere from a few Kb to 10Mb.

Regards,
Texernie1

 

[Stephen Lebans]

I would read the entire file into a single dimensioned Byte array and then
parse and convert to string data it from there. It would yield the fastest
processing times.
--

HTH
Stephen Lebans
http://www.lebans.com
Access Code, Tips and Tricks
Please respond only to the newsgroups so everyone can benefit.