Reading text data to double crlf

[amyl]

I have an application that is reading data from text files to the first
occurrence of a double crlf. I have this working using streamreader
and using readline.

However, I am hitting 500+ files at a time and this is a bit slow.

I was thinking of testing this using FileStream or something similar,
but I am not sure how to detect the occurrence of crlf crlf?

Thoughts?
Amy

 

[intrader]

I have an application that is reading data from text files to the first
occurrence of a double crlf. I have this working using streamreader
and using readline.

However, I am hitting 500+ files at a time and this is a bit slow.

I was thinking of testing this using FileStream or something similar,
but I am not sure how to detect the occurrence of crlf crlf?

Thoughts?
Amy

In the studio there is a configuration setting. Perhaps that is accessible
elsewhere?

 

[james.curran]

It's doubtful switch to FileStream will make it any faster.
StreamReader, when constructed using a file name, just creates a
FileStream internally. Your test would be just duplicating
StreamReader.

How are you reading it now (using streamreader & readline)?

 

[Kevin Spencer]

Hi Amy,

A couple of questions first, if I may:

1. Are these files always using the same encoding (Unicode, UTF8, ASCII,
etc)? It makes a difference as you can handle different encodings in
different ways. For example, ASCII is always 1 byte per character, whereas
Unicode is always 2 bytes per character, and needs a bit more effort to
read.

2. What is the expected (and projected) maximum size of any of these files?
You say that your app is processing 500+ at a time, but not what the maximum
size of any of these files is. If the files are small enough, you can read
each file all at once, rather than using ReadLine. Then you can simply
search the resulting string for "\r\n\r\n".

--
HTH,

Kevin Spencer
Microsoft MVP
Professional Numbskull

Hard work is a medication for which
there is no placebo.

 

[amyl]

It's doubtful switch to FileStream will make it any faster.
StreamReader, when constructed using a file name, just creates a
FileStream internally. Your test would be just duplicating
StreamReader.

How are you reading it now (using streamreader & readline)?

Yes, streamreader and readline.

It seems the file reading and parsing is fast. I beleive my major
delay is the listview being built.

Amy

 

[amyl]

1.) No, the encoding could be different.
2.) File size is random. Most are less < 20k, some could be as large
as 10mb.

Currently I am going line by line and stopping at the second \r\n\.

It seems my issue is not the reading of the files, but the listview
being created. It takes several seconds (5-10) to add about 1000
entries into a 6 column listview. Unfortuantly it still happens even
when using BeginUpdate and EndUpdate. Are there any other listview
tricks?

Amy

 

[Jon Skeet [C# MVP]]

1.) No, the encoding could be different.
2.) File size is random. Most are less < 20k, some could be as large
as 10mb.

Currently I am going line by line and stopping at the second \r\n\.

It seems my issue is not the reading of the files, but the listview
being created. It takes several seconds (5-10) to add about 1000
entries into a 6 column listview. Unfortuantly it still happens even
when using BeginUpdate and EndUpdate. Are there any other listview
tricks?

You should read the file in another thread and update the UI using
BeginInvoke.

See http://www.pobox.com/~skeet/csharp/threads/winforms.shtml

 

[james.curran]

How are you reading it now (using streamreader & readline)?
That wasn't intended as a yes/no question.

Rephrasing: "Your current method (the one that uses streamreader &
readline), what is it?"

However, your next statement would make that irrelevant.

 

[amyl]

Thanks Jon,

I am currently going this avenue. I am using .Net 2.0 and seen they
now have a BackgroundWorker. Have you used this class yet?

Amy.

 

[Jon Skeet [C# MVP]]

I am currently going this avenue. I am using .Net 2.0 and seen they
now have a BackgroundWorker. Have you used this class yet?

I haven't used it much, but that would do the trick, certainly. You'd
still need to be careful not to manipulate the UI directly from the
wrong thread. Read the docs carefully and you should be okay.