FileStream.Write slow to write last 100K of 2GB file

[Eric Cadwell]

Is there a faster way to write the last 100K of a large file? This code
takes almost two minutes to run on my machine.

int buffer = 100000;
int length = 2000000000;
string file = @"C:\test.txt";

if (File.Exists(file))
File.Delete(file);

int s = Environment.TickCount;
string test = "".PadLeft(buffer, 'a');
Byte[] data = Encoding.ASCII.GetBytes(test);
FileStream fs = File.Open(file, FileMode.OpenOrCreate);

int s2 = Environment.TickCount;
fs.SetLength(length);
System.Diagnostics.Debug.WriteLine("set: " + (Environment.TickCount - s2));

s2 = Environment.TickCount;
fs.Seek(length - buffer, SeekOrigin.Begin);
System.Diagnostics.Debug.WriteLine("seek: " + (Environment.TickCount - s2));

s2 = Environment.TickCount;
fs.Write(data, 0, buffer);
System.Diagnostics.Debug.WriteLine("write: " + (Environment.TickCount -
s2));

s2 = Environment.TickCount;
fs.Flush();
System.Diagnostics.Debug.WriteLine("flush: " + (Environment.TickCount -
s2));

fs.Close();
System.Diagnostics.Debug.WriteLine("complete: " + (Environment.TickCount -
s));


Thanks,
Eric

 

[TheSteph]

I tried your code... (Pentium IV 1,7 Ghz / 512Mb Ram / HD = IDE 7200rpm)



For a 100K file : set: 0 / seek: 0 / write: 0/ flush: 0 / complete: 15
(Immediate)

int buffer = 100000;

int length = 100005;



For a 2Gb File : set: 0 / seek: 0 / write: 40766 / flush: 0 / complete:
40781 (About 40Seconds)

int buffer = 100000;

int length = 2000000000;

Maybe the problem is your machine....

Steph.

 

[Eric Cadwell]

I'm running Centrino Duo 2Ghz, 2GB ram, 7200 RPM with Media Center Edition.
I would expect a millisecond response. If I write the file sequentially it's
sub-second all the way.

Thanks,
Eric

 

[Willy Denoyette [MVP]]

What you expect is not possible when writing to a newly created file.
When a file does not allready exists, all the file data must be written
starting from offset 0, so what you are measuring is effectively the time it
takes to write 2Gb of data.
If the file allready exists, writing 100Kb at the end will only take a
fraction of a second.
You can try this by commenting this part of your code when running a second
time:
if (File.Exists(file))
File.Delete(file);

Willy.




| Is there a faster way to write the last 100K of a large file? This code
| takes almost two minutes to run on my machine.
|
| int buffer = 100000;
| int length = 2000000000;
| string file = @"C:\test.txt";
|
| if (File.Exists(file))
| File.Delete(file);
|
| int s = Environment.TickCount;
| string test = "".PadLeft(buffer, 'a');
| Byte[] data = Encoding.ASCII.GetBytes(test);
| FileStream fs = File.Open(file, FileMode.OpenOrCreate);
|
| int s2 = Environment.TickCount;
| fs.SetLength(length);
| System.Diagnostics.Debug.WriteLine("set: " + (Environment.TickCount -
s2));
|
| s2 = Environment.TickCount;
| fs.Seek(length - buffer, SeekOrigin.Begin);
| System.Diagnostics.Debug.WriteLine("seek: " + (Environment.TickCount -
s2));
|
| s2 = Environment.TickCount;
| fs.Write(data, 0, buffer);
| System.Diagnostics.Debug.WriteLine("write: " + (Environment.TickCount -
| s2));
|
| s2 = Environment.TickCount;
| fs.Flush();
| System.Diagnostics.Debug.WriteLine("flush: " + (Environment.TickCount -
| s2));
|
| fs.Close();
| System.Diagnostics.Debug.WriteLine("complete: " + (Environment.TickCount -
| s));
|
|
| Thanks,
| Eric
|
|

 

[Eric Cadwell]

Thanks Willy,
Is there any way to interrupt it? If I want to close the application while
it's allocating space it takes two minutes to shut down.

-Eric

 

[Willy Denoyette [MVP]]

No, the file "creation" is a single IO action, and on current version of
Windows this cannot be canceled. The question is: why do you need to create
a 2GB file if you only wan't to write 100Kb real data to it?

Willy.


| Thanks Willy,
| Is there any way to interrupt it? If I want to close the application while
| it's allocating space it takes two minutes to shut down.
|
| -Eric
|
|