Streaming Techniques

[jp2msft]

What are the "Pros and Cons" with performing a stream operation in chunks as
opposed to performing that same operation on the full file?

"In Chunks" Example:
// *************************
void ReadFileInChunks(string file) {
using (FileStream fs = new FileStream(file, FileMode.Open,
FileAccess.Read)) {
int len;
byte[] buffer = new byte[1024];
do {
len = fs.Read(buffer, 0, 1024);
Console.WriteLine("Read 1024 bytes of data.");
} while (0 < len);
fs.Close();
}
}

"Full File" Example:
// *************************
void ReadFileAtOnce(string file) {
using (FileStream fs = new FileStream(file, FileMode.Open,
FileAccess.Read)) {
byte[] buffer = new byte[fs.Length];
if (fs.Read(buffer, 0, buffer.Length) == buffer.Length) {
Console.WriteLine("Finished!");
}
fs.Close();
}
}

I want to use a technique that will give me the greatest performance while
ensuring the method does not fail.

I'm guessing the ReadFileAtOnce method works fine as long as there is enough
RAM to read the entire file. (i.e. Passing ReadFileAtOnce a 6GB ZIP file
backup of a DVD would be bad.)

Any other thoughts on the subject?

 

[Nicholas Paldino [.NET/C# MVP]]

Well, first off, your examples are wrong. You are not guaranteed to get
the number of bytes back that you request, and should account for that in
your code.

That being said, if you have a very large file, it would probably be a
little prohibitive to load all of the data into memory. I'm being ambiguous
with terms like "very large file" and "prohibitive" because it depends on
the specs of your machine, the code, etc, etc. You will have to test to
determine which is best.

Reading the entire file if the file is 6GB probably would be fine on a
64 bit machine, but on a 32 bit machine, you are going to run into definite
issues.

 

[jp2msft]

So, if I were to read or write a little bit at a time, what is a good amout
to use? 1024 (1KB)?

Well, first off, your examples are wrong. You are not guaranteed to get
the number of bytes back that you request, and should account for that in
your code.

That being said, if you have a very large file, it would probably be a
little prohibitive to load all of the data into memory. I'm being ambiguous
with terms like "very large file" and "prohibitive" because it depends on
the specs of your machine, the code, etc, etc. You will have to test to
determine which is best.

Reading the entire file if the file is 6GB probably would be fine on a
64 bit machine, but on a 32 bit machine, you are going to run into definite
issues.

--
- Nicholas Paldino [.NET/C# MVP]
- (e-mail address removed)

What are the "Pros and Cons" with performing a stream operation in chunks
as
opposed to performing that same operation on the full file?

"In Chunks" Example:
// *************************
void ReadFileInChunks(string file) {
using (FileStream fs = new FileStream(file, FileMode.Open,
FileAccess.Read)) {
int len;
byte[] buffer = new byte[1024];
do {
len = fs.Read(buffer, 0, 1024);
Console.WriteLine("Read 1024 bytes of data.");
} while (0 < len);
fs.Close();
}
}

"Full File" Example:
// *************************
void ReadFileAtOnce(string file) {
using (FileStream fs = new FileStream(file, FileMode.Open,
FileAccess.Read)) {
byte[] buffer = new byte[fs.Length];
if (fs.Read(buffer, 0, buffer.Length) == buffer.Length) {
Console.WriteLine("Finished!");
}
fs.Close();
}
}

I want to use a technique that will give me the greatest performance while
ensuring the method does not fail.

I'm guessing the ReadFileAtOnce method works fine as long as there is
enough
RAM to read the entire file. (i.e. Passing ReadFileAtOnce a 6GB ZIP file
backup of a DVD would be bad.)

Any other thoughts on the subject?

 

[Nicholas Paldino [.NET/C# MVP]]

That depends completely on how much you need to process at a time, the
hardware of the machine you are runing on, and how much processing you have
to perform on each chunk.

You have to tweak and find out for yourself.

If your buffer is too small, you are going to have too many allocations
taking place, and that can affect performance. If the amount of memory
allocated is too large then you might have difficulties allocating the
space.

--
- Nicholas Paldino [.NET/C# MVP]
- (e-mail address removed)

So, if I were to read or write a little bit at a time, what is a good
amout
to use? 1024 (1KB)?

Well, first off, your examples are wrong. You are not guaranteed to
get
the number of bytes back that you request, and should account for that in
your code.

That being said, if you have a very large file, it would probably be
a
little prohibitive to load all of the data into memory. I'm being
ambiguous
with terms like "very large file" and "prohibitive" because it depends on
the specs of your machine, the code, etc, etc. You will have to test to
determine which is best.

Reading the entire file if the file is 6GB probably would be fine on
a
64 bit machine, but on a 32 bit machine, you are going to run into
definite
issues.

--
- Nicholas Paldino [.NET/C# MVP]
- (e-mail address removed)

What are the "Pros and Cons" with performing a stream operation in
chunks
as
opposed to performing that same operation on the full file?

"In Chunks" Example:
// *************************
void ReadFileInChunks(string file) {
using (FileStream fs = new FileStream(file, FileMode.Open,
FileAccess.Read)) {
int len;
byte[] buffer = new byte[1024];
do {
len = fs.Read(buffer, 0, 1024);
Console.WriteLine("Read 1024 bytes of data.");
} while (0 < len);
fs.Close();
}
}

"Full File" Example:
// *************************
void ReadFileAtOnce(string file) {
using (FileStream fs = new FileStream(file, FileMode.Open,
FileAccess.Read)) {
byte[] buffer = new byte[fs.Length];
if (fs.Read(buffer, 0, buffer.Length) == buffer.Length) {
Console.WriteLine("Finished!");
}
fs.Close();
}
}

I want to use a technique that will give me the greatest performance
while
ensuring the method does not fail.

I'm guessing the ReadFileAtOnce method works fine as long as there is
enough
RAM to read the entire file. (i.e. Passing ReadFileAtOnce a 6GB ZIP
file
backup of a DVD would be bad.)

Any other thoughts on the subject?

 

[Arne Vajhøj]

So, if I were to read or write a little bit at a time, what is a good amout
to use? 1024 (1KB)?

It used to be.

Today I would go for 10KB or 100KB.

Arne

 

[Jon Skeet [C# MVP]]

It used to be.

Today I would go for 10KB or 100KB.

There's a significant difference between these two, however - 100KB
ends up on the large object heap, which is never compacted (IIRC,
anyway). 10K ends up on the regular heap.

Personally I usually go for 8K or 16K; I'll try 32K or 64K for
benchmarking if appropriate; I wouldn't go above 64K without really
good evidence though.