Speeding up iterating through the filesystem on a file server.

[Peter Strøiman]

Hi.

I have an application that iterates through the file system of a file
server.
The application uses the classes DirectoryInfo and FileInfo to iterate
through
e.g.
void iterate( DirectoryInfo dir )
{
foreach ( DirectoryInfo subDir in dir.GetDirectories() )
{
iterate( subDir );
}
foreach ( FileInfo file in dir.GetFiles() )
{
doSomething ( file );
}
}

This takes quite a bit of time, when I have approx. 4Gb of data (many small
files)
Any ideas how I can speed it up?
I'm not an expert on code permission and security attributes. Are there any
attributes I can set in my program that would bypass code permission checks
( and only make them when starting up the app? )

Or are there quicker ways of obtaining the list of files and directories on
the server?

Thanks in advance,
Peter Strøiman

 

[Christine Nguyen]

This is just a guess.

I think you can speed things up by using 2 standard for loops instead of
foreach. If you use "foreach ( DirectoryInfo subDir in
dir.GetDirectories() )", your code has to perform a GetDirectories every
time it iterates through the loop, and this really slows it down. I usually
use VB.NET instead of C# so I'm not positive about the syntax but I think if
you do something like below, it will really speed things up. Let me know.

-Christine

int i;
int j;
DirectoryInfo[] diArr = di.GetDirectories();
int u = diArr.length -1;
DirectoryInfo subDir;
String fileArr[] = di.GetFiles;
Int k = fileArr.length - 1;
FileInfo file;

for(i=0; i <=u; i++){
subDir = diArr(i);
iterate(subDir);
for(j=0;j <= k; j++){
file = fileArr(j);
doSomething(file);
}
}

-Christine

 

[Jon Skeet]

This is just a guess.

I think you can speed things up by using 2 standard for loops instead of
foreach. If you use "foreach ( DirectoryInfo subDir in
dir.GetDirectories() )", your code has to perform a GetDirectories every
time it iterates through the loop, and this really slows it down.

No it doesn't. It executes GetDirectories once, then gets the
enumerator from that.

 

[Peter Strøiman]

Thanks for the reply.

First of all.

1: What I ( really ) do, is actually getting the array before iterating. I
simplified my example code.
2: It wouldn't call GetDirectories() for every iteration. Using the
foreach() syntax, the code will call GetDirectories() once and cast the
result to an IEnumerable.

Regards,
Peter Strøiman

This is just a guess.

I think you can speed things up by using 2 standard for loops instead of
foreach. If you use "foreach ( DirectoryInfo subDir in
dir.GetDirectories() )", your code has to perform a GetDirectories every
time it iterates through the loop, and this really slows it down. I usually
use VB.NET instead of C# so I'm not positive about the syntax but I think if
you do something like below, it will really speed things up. Let me know.

-Christine

int i;
int j;
DirectoryInfo[] diArr = di.GetDirectories();
int u = diArr.length -1;
DirectoryInfo subDir;
String fileArr[] = di.GetFiles;
Int k = fileArr.length - 1;
FileInfo file;

for(i=0; i <=u; i++){
subDir = diArr(i);
iterate(subDir);
for(j=0;j <= k; j++){
file = fileArr(j);
doSomething(file);
}
}

-Christine

Hi.

I have an application that iterates through the file system of a file
server.
The application uses the classes DirectoryInfo and FileInfo to iterate
through
e.g.
void iterate( DirectoryInfo dir )
{
foreach ( DirectoryInfo subDir in dir.GetDirectories() )
{
iterate( subDir );
}
foreach ( FileInfo file in dir.GetFiles() )
{
doSomething ( file );
}
}

This takes quite a bit of time, when I have approx. 4Gb of data (many small
files)
Any ideas how I can speed it up?
I'm not an expert on code permission and security attributes. Are there any
attributes I can set in my program that would bypass code permission checks
( and only make them when starting up the app? )

Or are there quicker ways of obtaining the list of files and directories on
the server?

Thanks in advance,
Peter Strøiman

 

[Günter Prossliner]

You're wrong. Calling for-each is allowed if the class implements
IEnumerable, witch has an method GetEnumerator() : IEnumerator. This
function is executed only once, when you enter the for-each. The IEnumerator
includes a state - machine for the loop.
You can speed up the code using an Index-Server Query.

GP

This is just a guess.

I think you can speed things up by using 2 standard for loops instead of
foreach. If you use "foreach ( DirectoryInfo subDir in
dir.GetDirectories() )", your code has to perform a GetDirectories every
time it iterates through the loop, and this really slows it down. I usually
use VB.NET instead of C# so I'm not positive about the syntax but I think if
you do something like below, it will really speed things up. Let me know.

-Christine

int i;
int j;
DirectoryInfo[] diArr = di.GetDirectories();
int u = diArr.length -1;
DirectoryInfo subDir;
String fileArr[] = di.GetFiles;
Int k = fileArr.length - 1;
FileInfo file;

for(i=0; i <=u; i++){
subDir = diArr(i);
iterate(subDir);
for(j=0;j <= k; j++){
file = fileArr(j);
doSomething(file);
}
}

-Christine

Hi.

I have an application that iterates through the file system of a file
server.
The application uses the classes DirectoryInfo and FileInfo to iterate
through
e.g.
void iterate( DirectoryInfo dir )
{
foreach ( DirectoryInfo subDir in dir.GetDirectories() )
{
iterate( subDir );
}
foreach ( FileInfo file in dir.GetFiles() )
{
doSomething ( file );
}
}

This takes quite a bit of time, when I have approx. 4Gb of data (many small
files)
Any ideas how I can speed it up?
I'm not an expert on code permission and security attributes. Are there any
attributes I can set in my program that would bypass code permission checks
( and only make them when starting up the app? )

Or are there quicker ways of obtaining the list of files and directories on
the server?

Thanks in advance,
Peter Strøiman

 

[Christine Nguyen]

All of you brought up really good points. I had forgotten about IEnumerator.
I've been able in the past to speed things up using this technique over
foreach, but maybe it was something particular to the collection I was
iterating.

This is just a guess.

I think you can speed things up by using 2 standard for loops instead of
foreach. If you use "foreach ( DirectoryInfo subDir in
dir.GetDirectories() )", your code has to perform a GetDirectories every
time it iterates through the loop, and this really slows it down. I usually
use VB.NET instead of C# so I'm not positive about the syntax but I think if
you do something like below, it will really speed things up. Let me know.

-Christine

int i;
int j;
DirectoryInfo[] diArr = di.GetDirectories();
int u = diArr.length -1;
DirectoryInfo subDir;
String fileArr[] = di.GetFiles;
Int k = fileArr.length - 1;
FileInfo file;

for(i=0; i <=u; i++){
subDir = diArr(i);
iterate(subDir);
for(j=0;j <= k; j++){
file = fileArr(j);
doSomething(file);
}
}

-Christine

Hi.

I have an application that iterates through the file system of a file
server.
The application uses the classes DirectoryInfo and FileInfo to iterate
through
e.g.
void iterate( DirectoryInfo dir )
{
foreach ( DirectoryInfo subDir in dir.GetDirectories() )
{
iterate( subDir );
}
foreach ( FileInfo file in dir.GetFiles() )
{
doSomething ( file );
}
}

This takes quite a bit of time, when I have approx. 4Gb of data (many small
files)
Any ideas how I can speed it up?
I'm not an expert on code permission and security attributes. Are there any
attributes I can set in my program that would bypass code permission checks
( and only make them when starting up the app? )

Or are there quicker ways of obtaining the list of files and directories on
the server?

Thanks in advance,
Peter Strøiman