Trouble with reading a text file spinning out of control

[Eric Kiernan]

I open and read a textfile ( 5500 lines ), check for conditions on each
line, and sometimes add ( 384 count )to a access mdb database; When I
run it, it would just go south and become unresponsive, and when i would
finally terminate it through task manager, there would be no records in
the table. I ended up putting the program to sleep peiodically as
below, and now everything works great. is this a known problem, or am i
doing something wrong?

StreamReader sr = new StreamReader(fs);
int loopCntr = 0;
while (sr.Peek() != -1) {
loopCntr++;
if ( loopCntr /100 == 0 ) {
Thread.Sleep(1);
} }

 

[Peter Duniho]

I open and read a textfile ( 5500 lines ), check for conditions on each
line, and sometimes add ( 384 count )to a access mdb database; When I
run it, it would just go south and become unresponsive, and when i would
finally terminate it through task manager, there would be no records in
the table. I ended up putting the program to sleep peiodically as
below, and now everything works great. is this a known problem, or am i
doing something wrong?

Surely you are doing something wrong. But without a concise-but-complete
codee example that reliably demonstrates the problem, there's no way for
any of us to know exactly what.

In general, polling is bad. If you must poll, polling without yielding
the CPU is very bad. But the OS won't let one thread hog the CPU
completely; eventually, other threads will be allowed to run. So things
eventually should complete. But yes, the code you posted is bad, and
would be even worse without the call to Thread.Sleep() (and really, you
should sleep with a value greater than 1 ms).

A much more appropriate way to monitor the file system for changes would
be to use the FileSystemWatcher class. No polling required.

Pete

 

[Jeff Johnson]

I open and read a textfile ( 5500 lines ), check for conditions on each
line, and sometimes add ( 384 count )to a access mdb database; When I run
it, it would just go south and become unresponsive, and when i would
finally terminate it through task manager, there would be no records in the
table. I ended up putting the program to sleep peiodically as below, and
now everything works great. is this a known problem, or am i doing
something wrong?

StreamReader sr = new StreamReader(fs);
int loopCntr = 0;
while (sr.Peek() != -1) {
loopCntr++;
if ( loopCntr /100 == 0 ) {
Thread.Sleep(1);
} }

The Peek() method does not advance the pointer through the file, so you're
reading the same character forever.

 

[Eric Kiernan]

I'm doing a sr.ReadLine just after that code. i just didn't want to
post a ton of code. but below is basically what happens

StreamReader sr = new StreamReader(fs);
string line;
while (sr.Peek() != -1) {
line = sr.ReadLine();
parse line;
if data from the line meets certain criteria, is is actually the first
field in a database record record, each line comprising a field.
save the data from the line to a structure with the same layout as the
table. because a record is composed of 4 lines. do sr.peeks and
sr.ReadLines 3 more times. Each ReadLine is evaluated to make sure it
meets criteria, and if not, a continue is issued. if ok, Then write the
4 variables to the 4 database fields.

if it doesn't meet criteria, do a continue to jump to the beginning of
the loop, where anoter sr.ReadLine will be performed if the peek is ok.

end while

 

[Eric Kiernan]

read my next post, i think i make it a little clearer. it is in pseudo
code. what is polling? is it me checking the count? i could post the
whole code, but it is 300 lines long. and uncommented mostly because it
is new code. it parses a quicken data exported qif file.

 

[Peter Duniho]

read my next post, i think i make it a little clearer.

Not really. I didn't find your follow-up post adding any new useful
information to the question at all.

it is in pseudo code. what is polling?
http://en.wikipedia.org/wiki/Polling_(computer_science)

is it me checking the count? i could post the whole code, but it is 300
lines long. and uncommented mostly because it is new code. it parses a
quicken data exported qif file.

I didn't suggest posting the whole code. I pointed out that without a
concise-but-complete code example, it's not possible to know exactly
what's wrong.

That said, my previous advice still stands. I suspect if you fix the code
so that it's not polling, you'll find things work better. Barring that,
just yield the CPU regularly.

Pete

 

[henry.lee.jr]

Not really.  I didn't find your follow-up post adding any new useful  
information to the question at all.


I didn't suggest posting the whole code.  I pointed out that without a  
concise-but-complete code example, it's not possible to know exactly  
what's wrong.

That said, my previous advice still stands.  I suspect if you fix the code  
so that it's not polling, you'll find things work better.  Barring that,  
just yield the CPU regularly.

Pete

Eric, I have to think there is an easier way to handle some of this?

I mean, at a minimum you should probably poll the file last modified
date as a starting point (instead of constantly reading the file) for
efficiency. As the previous poster mentioned, you really shouldn't be
sleeping for 1ms, because that is such a small sleep cycle that you
are almost pegging the cpu inadvertently. Unless this is a real-time
NASDAQ trading system, you can probably get away for sleeping for
anywhere from 1-10 seconds per poll. You could also start with the
FileSystemWatcher class to look for a /file change/ and trigger your
code, as opposed to your code constantly reading file IO.