Binary file writing problem

[Joshua Russell]

Hi,
I've got a program (see source below) that makes a file and fills it with
random binary values (from 0 to 255). The source below works, however the
program creates files at a rate of about 0.5MB per second. There is a
serious performance issue with this program.
There is a loop within the main loop that generates a buffer of 500 Bytes
that are then all written at once.
Can anybody tell me how I can improve the performance of the program? Idealy
I could with a rate of about 25Mb/s.

Thanx Josh



('reps' is the number of repetitions. e.g. Number of bytes)

FileStream fs = new FileStream(fileLocation, FileMode.CreateNew);

// Create the writer for data.

BinaryWriter w = new BinaryWriter(fs);



Random r =new Random();



for (int i = 0; i < Int32.Parse(reps.ToString()); )

{

byte[] bits = new byte[500];

for (int j = 0; j <= 499; j++)

{

bits[j] = byte.Parse(r.Next(0, 255).ToString());

i++;

if(i >= Int32.Parse(reps.ToString()))

{

break;

}

}

w.Write(bits);

}

 

[Willy Denoyette [MVP]]

Not sure what disk IO subsystem you are running this on, but don't ever
expect to achieve 25MB write throughput on anything less than a RAID0 (HW)
with a minimum of 7-8 spindles.
As a simple rule take a max. sustained IO rate of ~80-100 IO writes per
second per (7200 RPM) spindle.
So 500Kb per sec. is a fair number on a single drive with a block size of
500 bytes, what you should try first is to increase the buffer size to
something like 4-8 Kb, but don't expect higher throughputs than 2 - 3 Mb per
sec.

Willy.



source below) that makes a file and fills it with

 

[Sherif ElMetainy]

Hello

for (int i = 0; i < Int32.Parse(reps.ToString()); )

I understand that reps is a variable of type int. why convert it to a string
and back to int again? and this takes place at every iteration.
for(int i = 0; i < reps; i++) will do

bits[j] = byte.Parse(r.Next(0, 255).ToString());

Again converting to string and back to byte the following is much better.
bits[j] = (byte)r.Next(0, 255);

i++;
if(i >= Int32.Parse(reps.ToString()))
{
break;
}

The above is not needed, the check is redundant since you do the check in
the for statement, and again you convert to string and parse the string.
Also although i++ would work here, but its most appropriate place would be
in the for loop itself.

Converting an integer to a string includes allocation of a new string, and
the process writing the string representation of int in the allocated
string. The allocation will make garbage collections occur more frequently.
Converting from string to int is parsing each character in the string, and
checking if it is a valid number and calculating the resulting string. These
2 steps harm the performance badly in your case, and

The bits array is allocated reps times, I would allocate it once outside the
outer loop and reuse it to avoid allocations and garbage collections. Also
the BinaryReader itself is not usefull in your case, because you are writing
a byte array which can be done with FileStream. And since BinaryReader and
FileStream classes provides buffers, so you don't need your own buffer. You
can use WriteByte and FileStream will take care of buffering.

The program would be

FileStream fs = new FileStream(fileLocation, FileMode.CreateNew);
Random r =new Random();
int iterations = reps * 500;
for (int i = 0; i < iterations;i++ )
{
fs.WriteByte((byte)r.Next(0, 255));
}

 

[Willy Denoyette [MVP]]

Forget this, as Sherif correctly states the Filestream is buffered, so you
shouldn't buffer yourself.
Applying the changes as pointed out by Sherif, a sustained transfer rate of
10MB/sec should be possible on a single drive.
25 MB will however require a RAID0 stripe set of 2-3 drives.

Willy.