Large TXT Files

[Guest]

Hello All
i am trying to read a large txt files -may 1 GB-, which coze my program to
hang, and i need to know if there are techniques that make this without hang
and do it faster
if you know something can help. please tell me.

that is my code , and i had run it for a txt file its size is 423 MB, and i
left my PC opened and in the next day i found that it records about 10 hours
and large number of lines about 500000 line and also i got an exception
"OutOfMemoryException"
and i dont know much about "backgroundworker", if you know a complete
example plz provid me by the link

StreamReader SR;
private void btnReadFile_Click(object sender, EventArgs e)
{
DateTime DT= DateTime.Now;
TimeSpan T = new TimeSpan(DT.Day, DT.Hour, DT.Minute, DT.Second);
if (SR != null)
{
while (!SR.EndOfStream)
{
try
{
string line = SR.ReadLine();
txtFileContent.AppendText(line);
txtFileContent.AppendText("\n");
txtFileContent.AppendText("\r");
int count = Convert.ToInt32(labNumOfLines.Text);
count++;
labNumOfLines.Text = count.ToString();
DateTime tempTime = DateTime.Now;
TimeSpan T2 = new TimeSpan(tempTime.Day, tempTime.Hour, tempTime.Minute,
tempTime.Second);
T2=T2.Subtract(T);
labTime.Text = T2.ToString();
this.Refresh();
}
catch(Exception ex)
{
MessageBox.Show(ex.Message);
}

}
}
}


Note:txtFileContent is a TextBox

 

[Jon Skeet [C# MVP]]

On Mar 28, 12:48 pm, Just close your eyes and see > i am trying to
read a large txt files -may 1 GB-, which coze my program to

hang, and i need to know if there are techniques that make this without hang
and do it faster
if you know something can help. please tell me.

1) Don't do the reading in the UI thread.
2) Don't try to display that much data in the UI - page it if
necessary
3) Don't constantly convert the count between string and int values.
(Just declare count with a wider scope and increment it for each
line.)

and i dont know much about "backgroundworker", if you know a complete
example plz provid me by the link

Have you looked on MSDN? There are usually examples there. Here's a
general page about WinForms threading:

http://pobox.com/~skeet/csharp/threads/winforms.shtml

Jon

 

[Peter Duniho]

[...]
that is my code , and i had run it for a txt file its size is 423 MB,
and i left my PC opened and in the next day i found that it records
about 10 hours and large number of lines about 500000 line and also
i got an exception "OutOfMemoryException"

The big problem is your expectation that you can read the entire file into
memory at once. The theoretical maximum for a single object in Win32 and
..NET is 2GB (the size of the virtual address space allocated to the
proces), but in reality you'll never get close to that because your
program has other stuff allocated in memory as well. Because of
fragmentation (especially a problem if you are continually reallocating an
object, as you are here with your "AppendText" calls), the maximum size
can be MUCH less than 2GB, or even much less than 1GB.

This is exactly why you have an "OutOfMemoryException". You have
literally run out of memory: there is not a large enough contiguous block
of virtual address space to contain the object txFileContent.

As Jon says, the only solution is to not try to contain the entire file's
data in memory at once. You'll have to process the file in smaller
parts. This will not only prevent the out-of-memory condition, the
processing of the file should be significantly improved (when you're
appending, every time the size of the object has to change, ALL of the
data has to be copied to a new block of memory...that copy gets slower and
slower as the object gets larger and larger).

In addition to that, and the other comments he provides, I'll point out a
very minor issue: there's no need for you to create a TimeSpan object at
the beginning. If you simply store the "Now" DateTime in DT at the
beginning, you can subtract that from DateTime.Now later, and that
subtraction will result in a TimeSpan object.

Pete