Scanning hard drive

R

Rotsey

Hi,

I am writing an app that scans hard drives and logs info
about every fine on the drive.

The first iteration of my code used a class and a generic list
to store the data and rhis took 13min on my 60 GB drive.

I wanted it to be quicker.

So the second time I changed the class to a struct and still
with the generic list got the time down to 4 min.

I am wondering if I can improve on this even more??

Anyone know if it would be possible to improve this
using C# or would it need to be wrritten in assembler or the like?

rotsey
 
M

Marc Gravell

It depends entirely on how you are doing the writing code! 13 or 4
mins sounds big - how much data?
 
W

Willy Denoyette [MVP]

Rotsey said:
Hi,

I am writing an app that scans hard drives and logs info
about every fine on the drive.

The first iteration of my code used a class and a generic list
to store the data and rhis took 13min on my 60 GB drive.

I wanted it to be quicker.

How much, what are your expectations? The running time will depend on the
number of folders and files and the amount of info you are storing in the
list.
So the second time I changed the class to a struct and still
with the generic list got the time down to 4 min.

This is normal, the "fileinfo" was cached in the filesystem cache with the
last run, this highly reduces the Filesystem IO rate to get at the file
information.
I am wondering if I can improve on this even more??
My guess is that your system starts paging because you are exhausting the
Free RAM space with the data in your list. So, you need to start looking at
your memory consumption, and change your design, for instance by serializing
the List as a File on disk per directory you have scanned.
Anyone know if it would be possible to improve this
using C# or would it need to be wrritten in assembler or the like?

No, this has nothing to do with the language, really.


Willy.
 
R

Rotsey

Interesting, I rebooted and ran it.

This time approx 10 mins to scan 56GB.

It seems you are write that the fileinfo was being cached.

I checked mem usage in task manager and it say 110MB
for the app after it ran.

Which is not a problem if you 2GB like me or even 1GB really.

So I don't see why it would have paging issues, do you think??

So you don't think I can improve on this??????

Malcolm
 
A

Alun Harford

Rotsey said:
Hi,

I am writing an app that scans hard drives and logs info
about every fine on the drive.

The first iteration of my code used a class and a generic list
to store the data and rhis took 13min on my 60 GB drive.

I wanted it to be quicker.

So the second time I changed the class to a struct and still
with the generic list got the time down to 4 min.

That's not because you've used a struct. It's because the data is cached
by the operating system. Restart your computer and try it again.

The problem is the speed that you can get the data from the disk. There
isn't a lot you can do about that (except buy a faster hard disk, or
delete data, or make sure your OS is caching the stuff)

Alun Harford
 
W

Willy Denoyette [MVP]

Rotsey said:
Interesting, I rebooted and ran it.

This time approx 10 mins to scan 56GB.

It seems you are write that the fileinfo was being cached.

I checked mem usage in task manager and it say 110MB
for the app after it ran.

You need to look while it runs, not after it ran, and you need to use
perfmon to watch the working set size and the private bytes.
Which is not a problem if you 2GB like me or even 1GB really.

So I don't see why it would have paging issues, do you think??

This shouldn't be an issue, unless you have a very large number of files
stored on disk and you are storing a lot of file properties in the list.
Most likely is that you are running this on a slow disk and/or system. Most
info must be retrieved from disk, this involves Disk IO , but > 10 minutes
to get at this info is really slow, even 4 minutes for a second run is quite
slow.

That said, you need to tell us how many files you have on disk, what data
you are storing in the list, what OS and system HW you are running this on,
and (preferably) post a complete sample that illustrates the issue.

Willy.
 
B

Bela Istok

What do you scan? I do a directory and File scan in my 100GB (64GB of data)
hard drive and took 24 seconds. (Scanned every directory and every file in
the drive for info).

PD: for reference 170,187 files and 21,899 folders.


Regards,

Bela Istok
 
R

Rotsey

OK. Here is the class the scans the drive.

Can you see any issues here?

Bela, would you be able to post your code that scans the drive.

I am scanning 56GB in about 10 minutes.
public struct PieFileInfo

{

public int RowID;

public PieInfoType Type;

public string Name;

public string FullName;

public double Filesize;

public DateTime LastAccess;

public DateTime Created;

public string Extension;

public DateTime LastWrite;

public int ParentFolderRowID;

public string Drive;

public double FileAge1;

public double FileAge2;

public double FileAge3;

public bool IsRoot;

}

public class ScanDrive

{

private DriveData mList = new DriveData();

public DriveData DataList

{

get { return mList; }

set { mList = value; }

}

private string mDrive;

private int mNewFolderID = 0;

public ScanDrive(string drivefolder)

{

mDrive = drivefolder;

mList.Drive = drivefolder;

}

public bool ScanFolder(string folder, int folderID, ref double psize1, ref
double psize2, ref double psize3)

{

DirectoryInfo objDir = new DirectoryInfo(folder);

int newFolderID;

double size1 = 0;

double size2 = 0;

double size3 = 0;

System.DateTime fileDate;

try

{

newFolderID = WriteRow(objDir, folderID);

if (newFolderID == 0)

{

return false;

}

foreach (DirectoryInfo dir in objDir.GetDirectories())

{

if (!ScanFolder(dir.FullName, newFolderID, ref psize1,ref psize2,ref
psize3))

{

return true;

}

size1 += psize1;

size2 += psize2;

size3 += psize3;

}

foreach (FileInfo fle in objDir.GetFiles())

{

WriteRow(fle, newFolderID);

fileDate = GetAgeDate(fle);

if (fileDate >=
DateTime.Today.AddDays(Settings.Instance.SettingsData.Age1Days * -1))

size1 += fle.Length;

else if (fileDate >=
DateTime.Today.AddDays(Settings.Instance.SettingsData.Age2Days * -1))

size2 += fle.Length;

else

size3 += fle.Length;

}

UpdateFolderRow(newFolderID, size1, size2, size3);

psize1 = size1;

psize2 = size2;

psize3 = size3;

return true;

}

catch (Exception)

{

return true;

}



}

private DateTime GetAgeDate(FileInfo f)

{

if (f.CreationTime >=
DateTime.Today.AddDays(Settings.Instance.SettingsData.Age1Days * -1))

return f.CreationTime;

else if (f.LastWriteTime >=
DateTime.Today.AddDays(Settings.Instance.SettingsData.Age2Days * -1))

return f.LastWriteTime;

else

return f.LastAccessTime;

}

private int WriteRow(DirectoryInfo dir, int folderID)

{

PieFileInfo info = new PieFileInfo();

if (mNewFolderID == 0)

info.IsRoot = true;

mNewFolderID++;

info.RowID = mNewFolderID;

info.Type = PieInfoType.Folder;

info.FullName = dir.FullName;

info.Name = dir.Name;

info.LastAccess = dir.LastAccessTime;

info.LastWrite = dir.LastWriteTime;

info.Created = dir.CreationTime;

info.ParentFolderRowID = folderID;

info.Drive = mDrive;

mList.DataList.Add(info);

return mNewFolderID;

}

private int WriteRow(FileInfo file, int folderID)

{

PieFileInfo info = new PieFileInfo();

info.Type = PieInfoType.File;

info.Name = file.Name;

info.FullName = file.FullName;

info.Filesize = file.Length;

info.Extension = file.Extension;

info.LastAccess = file.LastAccessTime;

info.LastWrite = file.LastWriteTime;

info.Created = file.CreationTime;

info.ParentFolderRowID = folderID;

info.Drive = mDrive;

if (file.LastAccessTime >= DateTime.Today.AddDays(-30))

info.FileAge1 = file.Length;

else if (file.LastAccessTime >= DateTime.Today.AddDays(-60))

info.FileAge2 = file.Length;

else

info.FileAge3 = file.Length;

mList.DataList.Add(info);

return folderID;

}

private void UpdateFolderRow(int folderID, double size1, double size2,
double size3)

{

int index = mList.DataList.FindIndex(delegate(PieFileInfo fi)

{

if (fi.Type == PieInfoType.Folder && fi.RowID == folderID)

return true;

else

return false;

});

PieFileInfo info = mList.DataList[index];

info.FileAge1 += size1;

info.FileAge2 += size2;

info.FileAge3 += size3;

info.Filesize += size1 + size2 + size3;

mList.DataList[index] = info;

}

}
 
S

Stephany Young

Approx. 13 seconds on a 250GB disk with 30,000+ directories and 1,000,000+
files.

What is DriveData? (You omitted to post it).

What do you do with the content of DriveData when the call to
ScanFolder(...) returns?



Rotsey said:
OK. Here is the class the scans the drive.

Can you see any issues here?

Bela, would you be able to post your code that scans the drive.

I am scanning 56GB in about 10 minutes.
public struct PieFileInfo

{

public int RowID;

public PieInfoType Type;

public string Name;

public string FullName;

public double Filesize;

public DateTime LastAccess;

public DateTime Created;

public string Extension;

public DateTime LastWrite;

public int ParentFolderRowID;

public string Drive;

public double FileAge1;

public double FileAge2;

public double FileAge3;

public bool IsRoot;

}

public class ScanDrive

{

private DriveData mList = new DriveData();

public DriveData DataList

{

get { return mList; }

set { mList = value; }

}

private string mDrive;

private int mNewFolderID = 0;

public ScanDrive(string drivefolder)

{

mDrive = drivefolder;

mList.Drive = drivefolder;

}

public bool ScanFolder(string folder, int folderID, ref double psize1, ref
double psize2, ref double psize3)

{

DirectoryInfo objDir = new DirectoryInfo(folder);

int newFolderID;

double size1 = 0;

double size2 = 0;

double size3 = 0;

System.DateTime fileDate;

try

{

newFolderID = WriteRow(objDir, folderID);

if (newFolderID == 0)

{

return false;

}

foreach (DirectoryInfo dir in objDir.GetDirectories())

{

if (!ScanFolder(dir.FullName, newFolderID, ref psize1,ref psize2,ref
psize3))

{

return true;

}

size1 += psize1;

size2 += psize2;

size3 += psize3;

}

foreach (FileInfo fle in objDir.GetFiles())

{

WriteRow(fle, newFolderID);

fileDate = GetAgeDate(fle);

if (fileDate >=
DateTime.Today.AddDays(Settings.Instance.SettingsData.Age1Days * -1))

size1 += fle.Length;

else if (fileDate >=
DateTime.Today.AddDays(Settings.Instance.SettingsData.Age2Days * -1))

size2 += fle.Length;

else

size3 += fle.Length;

}

UpdateFolderRow(newFolderID, size1, size2, size3);

psize1 = size1;

psize2 = size2;

psize3 = size3;

return true;

}

catch (Exception)

{

return true;

}



}

private DateTime GetAgeDate(FileInfo f)

{

if (f.CreationTime >=
DateTime.Today.AddDays(Settings.Instance.SettingsData.Age1Days * -1))

return f.CreationTime;

else if (f.LastWriteTime >=
DateTime.Today.AddDays(Settings.Instance.SettingsData.Age2Days * -1))

return f.LastWriteTime;

else

return f.LastAccessTime;

}

private int WriteRow(DirectoryInfo dir, int folderID)

{

PieFileInfo info = new PieFileInfo();

if (mNewFolderID == 0)

info.IsRoot = true;

mNewFolderID++;

info.RowID = mNewFolderID;

info.Type = PieInfoType.Folder;

info.FullName = dir.FullName;

info.Name = dir.Name;

info.LastAccess = dir.LastAccessTime;

info.LastWrite = dir.LastWriteTime;

info.Created = dir.CreationTime;

info.ParentFolderRowID = folderID;

info.Drive = mDrive;

mList.DataList.Add(info);

return mNewFolderID;

}

private int WriteRow(FileInfo file, int folderID)

{

PieFileInfo info = new PieFileInfo();

info.Type = PieInfoType.File;

info.Name = file.Name;

info.FullName = file.FullName;

info.Filesize = file.Length;

info.Extension = file.Extension;

info.LastAccess = file.LastAccessTime;

info.LastWrite = file.LastWriteTime;

info.Created = file.CreationTime;

info.ParentFolderRowID = folderID;

info.Drive = mDrive;

if (file.LastAccessTime >= DateTime.Today.AddDays(-30))

info.FileAge1 = file.Length;

else if (file.LastAccessTime >= DateTime.Today.AddDays(-60))

info.FileAge2 = file.Length;

else

info.FileAge3 = file.Length;

mList.DataList.Add(info);

return folderID;

}

private void UpdateFolderRow(int folderID, double size1, double size2,
double size3)

{

int index = mList.DataList.FindIndex(delegate(PieFileInfo fi)

{

if (fi.Type == PieInfoType.Folder && fi.RowID == folderID)

return true;

else

return false;

});

PieFileInfo info = mList.DataList[index];

info.FileAge1 += size1;

info.FileAge2 += size2;

info.FileAge3 += size3;

info.Filesize += size1 + size2 + size3;

mList.DataList[index] = info;

}

}



Bela Istok said:
What do you scan? I do a directory and File scan in my 100GB (64GB of
data) hard drive and took 24 seconds. (Scanned every directory and every
file in the drive for info).

PD: for reference 170,187 files and 21,899 folders.


Regards,

Bela Istok
 
R

Rotsey

Sorry about that , but it just a class with the generic list and a string
drive property

All I am doing is displaying some of the info when ScanFolder returns
But that is after the scan completes of course which is not the problem.

Any chance of seeing your code???

public class DriveData

{

private List<PieFileInfo> mList = new List<PieFileInfo>();

public List<PieFileInfo> DataList

{

get { return mList; }

set { mList = value; }

}

private string mDrive;

public string Drive

{

get { return mDrive; }

set { mDrive = value; }

}

}

Stephany Young said:
Approx. 13 seconds on a 250GB disk with 30,000+ directories and 1,000,000+
files.

What is DriveData? (You omitted to post it).

What do you do with the content of DriveData when the call to
ScanFolder(...) returns?



Rotsey said:
OK. Here is the class the scans the drive.

Can you see any issues here?

Bela, would you be able to post your code that scans the drive.

I am scanning 56GB in about 10 minutes.
public struct PieFileInfo

{

public int RowID;

public PieInfoType Type;

public string Name;

public string FullName;

public double Filesize;

public DateTime LastAccess;

public DateTime Created;

public string Extension;

public DateTime LastWrite;

public int ParentFolderRowID;

public string Drive;

public double FileAge1;

public double FileAge2;

public double FileAge3;

public bool IsRoot;

}

public class ScanDrive

{

private DriveData mList = new DriveData();

public DriveData DataList

{

get { return mList; }

set { mList = value; }

}

private string mDrive;

private int mNewFolderID = 0;

public ScanDrive(string drivefolder)

{

mDrive = drivefolder;

mList.Drive = drivefolder;

}

public bool ScanFolder(string folder, int folderID, ref double psize1,
ref double psize2, ref double psize3)

{

DirectoryInfo objDir = new DirectoryInfo(folder);

int newFolderID;

double size1 = 0;

double size2 = 0;

double size3 = 0;

System.DateTime fileDate;

try

{

newFolderID = WriteRow(objDir, folderID);

if (newFolderID == 0)

{

return false;

}

foreach (DirectoryInfo dir in objDir.GetDirectories())

{

if (!ScanFolder(dir.FullName, newFolderID, ref psize1,ref psize2,ref
psize3))

{

return true;

}

size1 += psize1;

size2 += psize2;

size3 += psize3;

}

foreach (FileInfo fle in objDir.GetFiles())

{

WriteRow(fle, newFolderID);

fileDate = GetAgeDate(fle);

if (fileDate >=
DateTime.Today.AddDays(Settings.Instance.SettingsData.Age1Days * -1))

size1 += fle.Length;

else if (fileDate >=
DateTime.Today.AddDays(Settings.Instance.SettingsData.Age2Days * -1))

size2 += fle.Length;

else

size3 += fle.Length;

}

UpdateFolderRow(newFolderID, size1, size2, size3);

psize1 = size1;

psize2 = size2;

psize3 = size3;

return true;

}

catch (Exception)

{

return true;

}



}

private DateTime GetAgeDate(FileInfo f)

{

if (f.CreationTime >=
DateTime.Today.AddDays(Settings.Instance.SettingsData.Age1Days * -1))

return f.CreationTime;

else if (f.LastWriteTime >=
DateTime.Today.AddDays(Settings.Instance.SettingsData.Age2Days * -1))

return f.LastWriteTime;

else

return f.LastAccessTime;

}

private int WriteRow(DirectoryInfo dir, int folderID)

{

PieFileInfo info = new PieFileInfo();

if (mNewFolderID == 0)

info.IsRoot = true;

mNewFolderID++;

info.RowID = mNewFolderID;

info.Type = PieInfoType.Folder;

info.FullName = dir.FullName;

info.Name = dir.Name;

info.LastAccess = dir.LastAccessTime;

info.LastWrite = dir.LastWriteTime;

info.Created = dir.CreationTime;

info.ParentFolderRowID = folderID;

info.Drive = mDrive;

mList.DataList.Add(info);

return mNewFolderID;

}

private int WriteRow(FileInfo file, int folderID)

{

PieFileInfo info = new PieFileInfo();

info.Type = PieInfoType.File;

info.Name = file.Name;

info.FullName = file.FullName;

info.Filesize = file.Length;

info.Extension = file.Extension;

info.LastAccess = file.LastAccessTime;

info.LastWrite = file.LastWriteTime;

info.Created = file.CreationTime;

info.ParentFolderRowID = folderID;

info.Drive = mDrive;

if (file.LastAccessTime >= DateTime.Today.AddDays(-30))

info.FileAge1 = file.Length;

else if (file.LastAccessTime >= DateTime.Today.AddDays(-60))

info.FileAge2 = file.Length;

else

info.FileAge3 = file.Length;

mList.DataList.Add(info);

return folderID;

}

private void UpdateFolderRow(int folderID, double size1, double size2,
double size3)

{

int index = mList.DataList.FindIndex(delegate(PieFileInfo fi)

{

if (fi.Type == PieInfoType.Folder && fi.RowID == folderID)

return true;

else

return false;

});

PieFileInfo info = mList.DataList[index];

info.FileAge1 += size1;

info.FileAge2 += size2;

info.FileAge3 += size3;

info.Filesize += size1 + size2 + size3;

mList.DataList[index] = info;

}

}



Bela Istok said:
What do you scan? I do a directory and File scan in my 100GB (64GB of
data) hard drive and took 24 seconds. (Scanned every directory and every
file in the drive for info).

PD: for reference 170,187 files and 21,899 folders.


Regards,

Bela Istok

Hi,

I am writing an app that scans hard drives and logs info
about every fine on the drive.

The first iteration of my code used a class and a generic list
to store the data and rhis took 13min on my 60 GB drive.

I wanted it to be quicker.

So the second time I changed the class to a struct and still
with the generic list got the time down to 4 min.

I am wondering if I can improve on this even more??

Anyone know if it would be possible to improve this
using C# or would it need to be wrritten in assembler or the like?

rotsey
 
S

Stephany Young

It was YOUR code!


Rotsey said:
Sorry about that , but it just a class with the generic list and a string
drive property

All I am doing is displaying some of the info when ScanFolder returns
But that is after the scan completes of course which is not the problem.

Any chance of seeing your code???

public class DriveData

{

private List<PieFileInfo> mList = new List<PieFileInfo>();

public List<PieFileInfo> DataList

{

get { return mList; }

set { mList = value; }

}

private string mDrive;

public string Drive

{

get { return mDrive; }

set { mDrive = value; }

}

}

Stephany Young said:
Approx. 13 seconds on a 250GB disk with 30,000+ directories and
1,000,000+ files.

What is DriveData? (You omitted to post it).

What do you do with the content of DriveData when the call to
ScanFolder(...) returns?



Rotsey said:
OK. Here is the class the scans the drive.

Can you see any issues here?

Bela, would you be able to post your code that scans the drive.

I am scanning 56GB in about 10 minutes.
public struct PieFileInfo

{

public int RowID;

public PieInfoType Type;

public string Name;

public string FullName;

public double Filesize;

public DateTime LastAccess;

public DateTime Created;

public string Extension;

public DateTime LastWrite;

public int ParentFolderRowID;

public string Drive;

public double FileAge1;

public double FileAge2;

public double FileAge3;

public bool IsRoot;

}

public class ScanDrive

{

private DriveData mList = new DriveData();

public DriveData DataList

{

get { return mList; }

set { mList = value; }

}

private string mDrive;

private int mNewFolderID = 0;

public ScanDrive(string drivefolder)

{

mDrive = drivefolder;

mList.Drive = drivefolder;

}

public bool ScanFolder(string folder, int folderID, ref double psize1,
ref double psize2, ref double psize3)

{

DirectoryInfo objDir = new DirectoryInfo(folder);

int newFolderID;

double size1 = 0;

double size2 = 0;

double size3 = 0;

System.DateTime fileDate;

try

{

newFolderID = WriteRow(objDir, folderID);

if (newFolderID == 0)

{

return false;

}

foreach (DirectoryInfo dir in objDir.GetDirectories())

{

if (!ScanFolder(dir.FullName, newFolderID, ref psize1,ref psize2,ref
psize3))

{

return true;

}

size1 += psize1;

size2 += psize2;

size3 += psize3;

}

foreach (FileInfo fle in objDir.GetFiles())

{

WriteRow(fle, newFolderID);

fileDate = GetAgeDate(fle);

if (fileDate >=
DateTime.Today.AddDays(Settings.Instance.SettingsData.Age1Days * -1))

size1 += fle.Length;

else if (fileDate >=
DateTime.Today.AddDays(Settings.Instance.SettingsData.Age2Days * -1))

size2 += fle.Length;

else

size3 += fle.Length;

}

UpdateFolderRow(newFolderID, size1, size2, size3);

psize1 = size1;

psize2 = size2;

psize3 = size3;

return true;

}

catch (Exception)

{

return true;

}



}

private DateTime GetAgeDate(FileInfo f)

{

if (f.CreationTime >=
DateTime.Today.AddDays(Settings.Instance.SettingsData.Age1Days * -1))

return f.CreationTime;

else if (f.LastWriteTime >=
DateTime.Today.AddDays(Settings.Instance.SettingsData.Age2Days * -1))

return f.LastWriteTime;

else

return f.LastAccessTime;

}

private int WriteRow(DirectoryInfo dir, int folderID)

{

PieFileInfo info = new PieFileInfo();

if (mNewFolderID == 0)

info.IsRoot = true;

mNewFolderID++;

info.RowID = mNewFolderID;

info.Type = PieInfoType.Folder;

info.FullName = dir.FullName;

info.Name = dir.Name;

info.LastAccess = dir.LastAccessTime;

info.LastWrite = dir.LastWriteTime;

info.Created = dir.CreationTime;

info.ParentFolderRowID = folderID;

info.Drive = mDrive;

mList.DataList.Add(info);

return mNewFolderID;

}

private int WriteRow(FileInfo file, int folderID)

{

PieFileInfo info = new PieFileInfo();

info.Type = PieInfoType.File;

info.Name = file.Name;

info.FullName = file.FullName;

info.Filesize = file.Length;

info.Extension = file.Extension;

info.LastAccess = file.LastAccessTime;

info.LastWrite = file.LastWriteTime;

info.Created = file.CreationTime;

info.ParentFolderRowID = folderID;

info.Drive = mDrive;

if (file.LastAccessTime >= DateTime.Today.AddDays(-30))

info.FileAge1 = file.Length;

else if (file.LastAccessTime >= DateTime.Today.AddDays(-60))

info.FileAge2 = file.Length;

else

info.FileAge3 = file.Length;

mList.DataList.Add(info);

return folderID;

}

private void UpdateFolderRow(int folderID, double size1, double size2,
double size3)

{

int index = mList.DataList.FindIndex(delegate(PieFileInfo fi)

{

if (fi.Type == PieInfoType.Folder && fi.RowID == folderID)

return true;

else

return false;

});

PieFileInfo info = mList.DataList[index];

info.FileAge1 += size1;

info.FileAge2 += size2;

info.FileAge3 += size3;

info.Filesize += size1 + size2 + size3;

mList.DataList[index] = info;

}

}



What do you scan? I do a directory and File scan in my 100GB (64GB of
data) hard drive and took 24 seconds. (Scanned every directory and
every file in the drive for info).

PD: for reference 170,187 files and 21,899 folders.


Regards,

Bela Istok

Hi,

I am writing an app that scans hard drives and logs info
about every fine on the drive.

The first iteration of my code used a class and a generic list
to store the data and rhis took 13min on my 60 GB drive.

I wanted it to be quicker.

So the second time I changed the class to a struct and still
with the generic list got the time down to 4 min.

I am wondering if I can improve on this even more??

Anyone know if it would be possible to improve this
using C# or would it need to be wrritten in assembler or the like?

rotsey
 
R

Rotsey

Sorry Stephen what do you mean?



Stephany Young said:
It was YOUR code!


Rotsey said:
Sorry about that , but it just a class with the generic list and a string
drive property

All I am doing is displaying some of the info when ScanFolder returns
But that is after the scan completes of course which is not the problem.

Any chance of seeing your code???

public class DriveData

{

private List<PieFileInfo> mList = new List<PieFileInfo>();

public List<PieFileInfo> DataList

{

get { return mList; }

set { mList = value; }

}

private string mDrive;

public string Drive

{

get { return mDrive; }

set { mDrive = value; }

}

}

Stephany Young said:
Approx. 13 seconds on a 250GB disk with 30,000+ directories and
1,000,000+ files.

What is DriveData? (You omitted to post it).

What do you do with the content of DriveData when the call to
ScanFolder(...) returns?



OK. Here is the class the scans the drive.

Can you see any issues here?

Bela, would you be able to post your code that scans the drive.

I am scanning 56GB in about 10 minutes.
public struct PieFileInfo

{

public int RowID;

public PieInfoType Type;

public string Name;

public string FullName;

public double Filesize;

public DateTime LastAccess;

public DateTime Created;

public string Extension;

public DateTime LastWrite;

public int ParentFolderRowID;

public string Drive;

public double FileAge1;

public double FileAge2;

public double FileAge3;

public bool IsRoot;

}

public class ScanDrive

{

private DriveData mList = new DriveData();

public DriveData DataList

{

get { return mList; }

set { mList = value; }

}

private string mDrive;

private int mNewFolderID = 0;

public ScanDrive(string drivefolder)

{

mDrive = drivefolder;

mList.Drive = drivefolder;

}

public bool ScanFolder(string folder, int folderID, ref double psize1,
ref double psize2, ref double psize3)

{

DirectoryInfo objDir = new DirectoryInfo(folder);

int newFolderID;

double size1 = 0;

double size2 = 0;

double size3 = 0;

System.DateTime fileDate;

try

{

newFolderID = WriteRow(objDir, folderID);

if (newFolderID == 0)

{

return false;

}

foreach (DirectoryInfo dir in objDir.GetDirectories())

{

if (!ScanFolder(dir.FullName, newFolderID, ref psize1,ref psize2,ref
psize3))

{

return true;

}

size1 += psize1;

size2 += psize2;

size3 += psize3;

}

foreach (FileInfo fle in objDir.GetFiles())

{

WriteRow(fle, newFolderID);

fileDate = GetAgeDate(fle);

if (fileDate >=
DateTime.Today.AddDays(Settings.Instance.SettingsData.Age1Days * -1))

size1 += fle.Length;

else if (fileDate >=
DateTime.Today.AddDays(Settings.Instance.SettingsData.Age2Days * -1))

size2 += fle.Length;

else

size3 += fle.Length;

}

UpdateFolderRow(newFolderID, size1, size2, size3);

psize1 = size1;

psize2 = size2;

psize3 = size3;

return true;

}

catch (Exception)

{

return true;

}



}

private DateTime GetAgeDate(FileInfo f)

{

if (f.CreationTime >=
DateTime.Today.AddDays(Settings.Instance.SettingsData.Age1Days * -1))

return f.CreationTime;

else if (f.LastWriteTime >=
DateTime.Today.AddDays(Settings.Instance.SettingsData.Age2Days * -1))

return f.LastWriteTime;

else

return f.LastAccessTime;

}

private int WriteRow(DirectoryInfo dir, int folderID)

{

PieFileInfo info = new PieFileInfo();

if (mNewFolderID == 0)

info.IsRoot = true;

mNewFolderID++;

info.RowID = mNewFolderID;

info.Type = PieInfoType.Folder;

info.FullName = dir.FullName;

info.Name = dir.Name;

info.LastAccess = dir.LastAccessTime;

info.LastWrite = dir.LastWriteTime;

info.Created = dir.CreationTime;

info.ParentFolderRowID = folderID;

info.Drive = mDrive;

mList.DataList.Add(info);

return mNewFolderID;

}

private int WriteRow(FileInfo file, int folderID)

{

PieFileInfo info = new PieFileInfo();

info.Type = PieInfoType.File;

info.Name = file.Name;

info.FullName = file.FullName;

info.Filesize = file.Length;

info.Extension = file.Extension;

info.LastAccess = file.LastAccessTime;

info.LastWrite = file.LastWriteTime;

info.Created = file.CreationTime;

info.ParentFolderRowID = folderID;

info.Drive = mDrive;

if (file.LastAccessTime >= DateTime.Today.AddDays(-30))

info.FileAge1 = file.Length;

else if (file.LastAccessTime >= DateTime.Today.AddDays(-60))

info.FileAge2 = file.Length;

else

info.FileAge3 = file.Length;

mList.DataList.Add(info);

return folderID;

}

private void UpdateFolderRow(int folderID, double size1, double size2,
double size3)

{

int index = mList.DataList.FindIndex(delegate(PieFileInfo fi)

{

if (fi.Type == PieInfoType.Folder && fi.RowID == folderID)

return true;

else

return false;

});

PieFileInfo info = mList.DataList[index];

info.FileAge1 += size1;

info.FileAge2 += size2;

info.FileAge3 += size3;

info.Filesize += size1 + size2 + size3;

mList.DataList[index] = info;

}

}



What do you scan? I do a directory and File scan in my 100GB (64GB of
data) hard drive and took 24 seconds. (Scanned every directory and
every file in the drive for info).

PD: for reference 170,187 files and 21,899 folders.


Regards,

Bela Istok

Hi,

I am writing an app that scans hard drives and logs info
about every fine on the drive.

The first iteration of my code used a class and a generic list
to store the data and rhis took 13min on my 60 GB drive.

I wanted it to be quicker.

So the second time I changed the class to a struct and still
with the generic list got the time down to 4 min.

I am wondering if I can improve on this even more??

Anyone know if it would be possible to improve this
using C# or would it need to be wrritten in assembler or the like?

rotsey
 
S

Stephany Young

It's Stephany!

You asked 'Any chance of seeing your code???'

I replied 'It was your code (that I ran)'.

The DriveData class did not add any significant overhead.

I used 10 for Age1Days and 20 for Age2Days.

Here are my results for my main dev machine (Vista).

Drive size1 size2 size3 objects elapsed time
objects per second
-----------------------------------------------------------------------------------------------
C:\ 7,406,454,214 1,166,404,747 45,787,333,348 167,342 1:57.0308662
1429.8962780812093
D:\ 395,375,683 582,398,194 31,460,898,868 59,563 0:11.8571418
5023.3859900368232
E:\ 168 0 87,079,722,334 36,356 0:04.6710676
7783.2313966083470
-----------------------------------------------------------------------------------------------
7,801,830,065 1,748,802,941 164,328,154,550 263,461 2:13.5590756
1972.6177260244529

Grand Total: 173,878,787,556 (161.937240 GB)

C:\ drive is the system drive and has the paging file so those 2 aspects go
someway to explaining the slower rate.

Quite acceptable!


Rotsey said:
Sorry Stephen what do you mean?



Stephany Young said:
It was YOUR code!


Rotsey said:
Sorry about that , but it just a class with the generic list and a
string drive property

All I am doing is displaying some of the info when ScanFolder returns
But that is after the scan completes of course which is not the problem.

Any chance of seeing your code???

public class DriveData

{

private List<PieFileInfo> mList = new List<PieFileInfo>();

public List<PieFileInfo> DataList

{

get { return mList; }

set { mList = value; }

}

private string mDrive;

public string Drive

{

get { return mDrive; }

set { mDrive = value; }

}

}

Approx. 13 seconds on a 250GB disk with 30,000+ directories and
1,000,000+ files.

What is DriveData? (You omitted to post it).

What do you do with the content of DriveData when the call to
ScanFolder(...) returns?



OK. Here is the class the scans the drive.

Can you see any issues here?

Bela, would you be able to post your code that scans the drive.

I am scanning 56GB in about 10 minutes.
public struct PieFileInfo

{

public int RowID;

public PieInfoType Type;

public string Name;

public string FullName;

public double Filesize;

public DateTime LastAccess;

public DateTime Created;

public string Extension;

public DateTime LastWrite;

public int ParentFolderRowID;

public string Drive;

public double FileAge1;

public double FileAge2;

public double FileAge3;

public bool IsRoot;

}

public class ScanDrive

{

private DriveData mList = new DriveData();

public DriveData DataList

{

get { return mList; }

set { mList = value; }

}

private string mDrive;

private int mNewFolderID = 0;

public ScanDrive(string drivefolder)

{

mDrive = drivefolder;

mList.Drive = drivefolder;

}

public bool ScanFolder(string folder, int folderID, ref double psize1,
ref double psize2, ref double psize3)

{

DirectoryInfo objDir = new DirectoryInfo(folder);

int newFolderID;

double size1 = 0;

double size2 = 0;

double size3 = 0;

System.DateTime fileDate;

try

{

newFolderID = WriteRow(objDir, folderID);

if (newFolderID == 0)

{

return false;

}

foreach (DirectoryInfo dir in objDir.GetDirectories())

{

if (!ScanFolder(dir.FullName, newFolderID, ref psize1,ref psize2,ref
psize3))

{

return true;

}

size1 += psize1;

size2 += psize2;

size3 += psize3;

}

foreach (FileInfo fle in objDir.GetFiles())

{

WriteRow(fle, newFolderID);

fileDate = GetAgeDate(fle);

if (fileDate >=
DateTime.Today.AddDays(Settings.Instance.SettingsData.Age1Days * -1))

size1 += fle.Length;

else if (fileDate >=
DateTime.Today.AddDays(Settings.Instance.SettingsData.Age2Days * -1))

size2 += fle.Length;

else

size3 += fle.Length;

}

UpdateFolderRow(newFolderID, size1, size2, size3);

psize1 = size1;

psize2 = size2;

psize3 = size3;

return true;

}

catch (Exception)

{

return true;

}



}

private DateTime GetAgeDate(FileInfo f)

{

if (f.CreationTime >=
DateTime.Today.AddDays(Settings.Instance.SettingsData.Age1Days * -1))

return f.CreationTime;

else if (f.LastWriteTime >=
DateTime.Today.AddDays(Settings.Instance.SettingsData.Age2Days * -1))

return f.LastWriteTime;

else

return f.LastAccessTime;

}

private int WriteRow(DirectoryInfo dir, int folderID)

{

PieFileInfo info = new PieFileInfo();

if (mNewFolderID == 0)

info.IsRoot = true;

mNewFolderID++;

info.RowID = mNewFolderID;

info.Type = PieInfoType.Folder;

info.FullName = dir.FullName;

info.Name = dir.Name;

info.LastAccess = dir.LastAccessTime;

info.LastWrite = dir.LastWriteTime;

info.Created = dir.CreationTime;

info.ParentFolderRowID = folderID;

info.Drive = mDrive;

mList.DataList.Add(info);

return mNewFolderID;

}

private int WriteRow(FileInfo file, int folderID)

{

PieFileInfo info = new PieFileInfo();

info.Type = PieInfoType.File;

info.Name = file.Name;

info.FullName = file.FullName;

info.Filesize = file.Length;

info.Extension = file.Extension;

info.LastAccess = file.LastAccessTime;

info.LastWrite = file.LastWriteTime;

info.Created = file.CreationTime;

info.ParentFolderRowID = folderID;

info.Drive = mDrive;

if (file.LastAccessTime >= DateTime.Today.AddDays(-30))

info.FileAge1 = file.Length;

else if (file.LastAccessTime >= DateTime.Today.AddDays(-60))

info.FileAge2 = file.Length;

else

info.FileAge3 = file.Length;

mList.DataList.Add(info);

return folderID;

}

private void UpdateFolderRow(int folderID, double size1, double size2,
double size3)

{

int index = mList.DataList.FindIndex(delegate(PieFileInfo fi)

{

if (fi.Type == PieInfoType.Folder && fi.RowID == folderID)

return true;

else

return false;

});

PieFileInfo info = mList.DataList[index];

info.FileAge1 += size1;

info.FileAge2 += size2;

info.FileAge3 += size3;

info.Filesize += size1 + size2 + size3;

mList.DataList[index] = info;

}

}



What do you scan? I do a directory and File scan in my 100GB (64GB of
data) hard drive and took 24 seconds. (Scanned every directory and
every file in the drive for info).

PD: for reference 170,187 files and 21,899 folders.


Regards,

Bela Istok

Hi,

I am writing an app that scans hard drives and logs info
about every fine on the drive.

The first iteration of my code used a class and a generic list
to store the data and rhis took 13min on my 60 GB drive.

I wanted it to be quicker.

So the second time I changed the class to a struct and still
with the generic list got the time down to 4 min.

I am wondering if I can improve on this even more??

Anyone know if it would be possible to improve this
using C# or would it need to be wrritten in assembler or the like?

rotsey
 
R

Rotsey

Oh i see you mean you ran my code on your machine.

surely it can't be that fast

I have a 7200RPM 60GB drive in 1.86GHz Pentium M Laptop

What is your specs?


Rotsey said:
Sorry Stephen what do you mean?



Stephany Young said:
It was YOUR code!


Rotsey said:
Sorry about that , but it just a class with the generic list and a
string drive property

All I am doing is displaying some of the info when ScanFolder returns
But that is after the scan completes of course which is not the problem.

Any chance of seeing your code???

public class DriveData

{

private List<PieFileInfo> mList = new List<PieFileInfo>();

public List<PieFileInfo> DataList

{

get { return mList; }

set { mList = value; }

}

private string mDrive;

public string Drive

{

get { return mDrive; }

set { mDrive = value; }

}

}

Approx. 13 seconds on a 250GB disk with 30,000+ directories and
1,000,000+ files.

What is DriveData? (You omitted to post it).

What do you do with the content of DriveData when the call to
ScanFolder(...) returns?



OK. Here is the class the scans the drive.

Can you see any issues here?

Bela, would you be able to post your code that scans the drive.

I am scanning 56GB in about 10 minutes.
public struct PieFileInfo

{

public int RowID;

public PieInfoType Type;

public string Name;

public string FullName;

public double Filesize;

public DateTime LastAccess;

public DateTime Created;

public string Extension;

public DateTime LastWrite;

public int ParentFolderRowID;

public string Drive;

public double FileAge1;

public double FileAge2;

public double FileAge3;

public bool IsRoot;

}

public class ScanDrive

{

private DriveData mList = new DriveData();

public DriveData DataList

{

get { return mList; }

set { mList = value; }

}

private string mDrive;

private int mNewFolderID = 0;

public ScanDrive(string drivefolder)

{

mDrive = drivefolder;

mList.Drive = drivefolder;

}

public bool ScanFolder(string folder, int folderID, ref double psize1,
ref double psize2, ref double psize3)

{

DirectoryInfo objDir = new DirectoryInfo(folder);

int newFolderID;

double size1 = 0;

double size2 = 0;

double size3 = 0;

System.DateTime fileDate;

try

{

newFolderID = WriteRow(objDir, folderID);

if (newFolderID == 0)

{

return false;

}

foreach (DirectoryInfo dir in objDir.GetDirectories())

{

if (!ScanFolder(dir.FullName, newFolderID, ref psize1,ref psize2,ref
psize3))

{

return true;

}

size1 += psize1;

size2 += psize2;

size3 += psize3;

}

foreach (FileInfo fle in objDir.GetFiles())

{

WriteRow(fle, newFolderID);

fileDate = GetAgeDate(fle);

if (fileDate >=
DateTime.Today.AddDays(Settings.Instance.SettingsData.Age1Days * -1))

size1 += fle.Length;

else if (fileDate >=
DateTime.Today.AddDays(Settings.Instance.SettingsData.Age2Days * -1))

size2 += fle.Length;

else

size3 += fle.Length;

}

UpdateFolderRow(newFolderID, size1, size2, size3);

psize1 = size1;

psize2 = size2;

psize3 = size3;

return true;

}

catch (Exception)

{

return true;

}



}

private DateTime GetAgeDate(FileInfo f)

{

if (f.CreationTime >=
DateTime.Today.AddDays(Settings.Instance.SettingsData.Age1Days * -1))

return f.CreationTime;

else if (f.LastWriteTime >=
DateTime.Today.AddDays(Settings.Instance.SettingsData.Age2Days * -1))

return f.LastWriteTime;

else

return f.LastAccessTime;

}

private int WriteRow(DirectoryInfo dir, int folderID)

{

PieFileInfo info = new PieFileInfo();

if (mNewFolderID == 0)

info.IsRoot = true;

mNewFolderID++;

info.RowID = mNewFolderID;

info.Type = PieInfoType.Folder;

info.FullName = dir.FullName;

info.Name = dir.Name;

info.LastAccess = dir.LastAccessTime;

info.LastWrite = dir.LastWriteTime;

info.Created = dir.CreationTime;

info.ParentFolderRowID = folderID;

info.Drive = mDrive;

mList.DataList.Add(info);

return mNewFolderID;

}

private int WriteRow(FileInfo file, int folderID)

{

PieFileInfo info = new PieFileInfo();

info.Type = PieInfoType.File;

info.Name = file.Name;

info.FullName = file.FullName;

info.Filesize = file.Length;

info.Extension = file.Extension;

info.LastAccess = file.LastAccessTime;

info.LastWrite = file.LastWriteTime;

info.Created = file.CreationTime;

info.ParentFolderRowID = folderID;

info.Drive = mDrive;

if (file.LastAccessTime >= DateTime.Today.AddDays(-30))

info.FileAge1 = file.Length;

else if (file.LastAccessTime >= DateTime.Today.AddDays(-60))

info.FileAge2 = file.Length;

else

info.FileAge3 = file.Length;

mList.DataList.Add(info);

return folderID;

}

private void UpdateFolderRow(int folderID, double size1, double size2,
double size3)

{

int index = mList.DataList.FindIndex(delegate(PieFileInfo fi)

{

if (fi.Type == PieInfoType.Folder && fi.RowID == folderID)

return true;

else

return false;

});

PieFileInfo info = mList.DataList[index];

info.FileAge1 += size1;

info.FileAge2 += size2;

info.FileAge3 += size3;

info.Filesize += size1 + size2 + size3;

mList.DataList[index] = info;

}

}



What do you scan? I do a directory and File scan in my 100GB (64GB of
data) hard drive and took 24 seconds. (Scanned every directory and
every file in the drive for info).

PD: for reference 170,187 files and 21,899 folders.


Regards,

Bela Istok

Hi,

I am writing an app that scans hard drives and logs info
about every fine on the drive.

The first iteration of my code used a class and a generic list
to store the data and rhis took 13min on my 60 GB drive.

I wanted it to be quicker.

So the second time I changed the class to a struct and still
with the generic list got the time down to 4 min.

I am wondering if I can improve on this even more??

Anyone know if it would be possible to improve this
using C# or would it need to be wrritten in assembler or the like?

rotsey
 
S

Stephany Young

The earlier run was on a super grunty server with really fast scsi drives,
truckloads of ram and multiple dual-core perocessors.


Rotsey said:
Oh i see you mean you ran my code on your machine.

surely it can't be that fast

I have a 7200RPM 60GB drive in 1.86GHz Pentium M Laptop

What is your specs?


Rotsey said:
Sorry Stephen what do you mean?



Stephany Young said:
It was YOUR code!


Sorry about that , but it just a class with the generic list and a
string drive property

All I am doing is displaying some of the info when ScanFolder returns
But that is after the scan completes of course which is not the
problem.

Any chance of seeing your code???

public class DriveData

{

private List<PieFileInfo> mList = new List<PieFileInfo>();

public List<PieFileInfo> DataList

{

get { return mList; }

set { mList = value; }

}

private string mDrive;

public string Drive

{

get { return mDrive; }

set { mDrive = value; }

}

}

Approx. 13 seconds on a 250GB disk with 30,000+ directories and
1,000,000+ files.

What is DriveData? (You omitted to post it).

What do you do with the content of DriveData when the call to
ScanFolder(...) returns?



OK. Here is the class the scans the drive.

Can you see any issues here?

Bela, would you be able to post your code that scans the drive.

I am scanning 56GB in about 10 minutes.
public struct PieFileInfo

{

public int RowID;

public PieInfoType Type;

public string Name;

public string FullName;

public double Filesize;

public DateTime LastAccess;

public DateTime Created;

public string Extension;

public DateTime LastWrite;

public int ParentFolderRowID;

public string Drive;

public double FileAge1;

public double FileAge2;

public double FileAge3;

public bool IsRoot;

}

public class ScanDrive

{

private DriveData mList = new DriveData();

public DriveData DataList

{

get { return mList; }

set { mList = value; }

}

private string mDrive;

private int mNewFolderID = 0;

public ScanDrive(string drivefolder)

{

mDrive = drivefolder;

mList.Drive = drivefolder;

}

public bool ScanFolder(string folder, int folderID, ref double
psize1, ref double psize2, ref double psize3)

{

DirectoryInfo objDir = new DirectoryInfo(folder);

int newFolderID;

double size1 = 0;

double size2 = 0;

double size3 = 0;

System.DateTime fileDate;

try

{

newFolderID = WriteRow(objDir, folderID);

if (newFolderID == 0)

{

return false;

}

foreach (DirectoryInfo dir in objDir.GetDirectories())

{

if (!ScanFolder(dir.FullName, newFolderID, ref psize1,ref psize2,ref
psize3))

{

return true;

}

size1 += psize1;

size2 += psize2;

size3 += psize3;

}

foreach (FileInfo fle in objDir.GetFiles())

{

WriteRow(fle, newFolderID);

fileDate = GetAgeDate(fle);

if (fileDate >=
DateTime.Today.AddDays(Settings.Instance.SettingsData.Age1Days * -1))

size1 += fle.Length;

else if (fileDate >=
DateTime.Today.AddDays(Settings.Instance.SettingsData.Age2Days * -1))

size2 += fle.Length;

else

size3 += fle.Length;

}

UpdateFolderRow(newFolderID, size1, size2, size3);

psize1 = size1;

psize2 = size2;

psize3 = size3;

return true;

}

catch (Exception)

{

return true;

}



}

private DateTime GetAgeDate(FileInfo f)

{

if (f.CreationTime >=
DateTime.Today.AddDays(Settings.Instance.SettingsData.Age1Days * -1))

return f.CreationTime;

else if (f.LastWriteTime >=
DateTime.Today.AddDays(Settings.Instance.SettingsData.Age2Days * -1))

return f.LastWriteTime;

else

return f.LastAccessTime;

}

private int WriteRow(DirectoryInfo dir, int folderID)

{

PieFileInfo info = new PieFileInfo();

if (mNewFolderID == 0)

info.IsRoot = true;

mNewFolderID++;

info.RowID = mNewFolderID;

info.Type = PieInfoType.Folder;

info.FullName = dir.FullName;

info.Name = dir.Name;

info.LastAccess = dir.LastAccessTime;

info.LastWrite = dir.LastWriteTime;

info.Created = dir.CreationTime;

info.ParentFolderRowID = folderID;

info.Drive = mDrive;

mList.DataList.Add(info);

return mNewFolderID;

}

private int WriteRow(FileInfo file, int folderID)

{

PieFileInfo info = new PieFileInfo();

info.Type = PieInfoType.File;

info.Name = file.Name;

info.FullName = file.FullName;

info.Filesize = file.Length;

info.Extension = file.Extension;

info.LastAccess = file.LastAccessTime;

info.LastWrite = file.LastWriteTime;

info.Created = file.CreationTime;

info.ParentFolderRowID = folderID;

info.Drive = mDrive;

if (file.LastAccessTime >= DateTime.Today.AddDays(-30))

info.FileAge1 = file.Length;

else if (file.LastAccessTime >= DateTime.Today.AddDays(-60))

info.FileAge2 = file.Length;

else

info.FileAge3 = file.Length;

mList.DataList.Add(info);

return folderID;

}

private void UpdateFolderRow(int folderID, double size1, double
size2, double size3)

{

int index = mList.DataList.FindIndex(delegate(PieFileInfo fi)

{

if (fi.Type == PieInfoType.Folder && fi.RowID == folderID)

return true;

else

return false;

});

PieFileInfo info = mList.DataList[index];

info.FileAge1 += size1;

info.FileAge2 += size2;

info.FileAge3 += size3;

info.Filesize += size1 + size2 + size3;

mList.DataList[index] = info;

}

}



What do you scan? I do a directory and File scan in my 100GB (64GB
of data) hard drive and took 24 seconds. (Scanned every directory
and every file in the drive for info).

PD: for reference 170,187 files and 21,899 folders.


Regards,

Bela Istok

Hi,

I am writing an app that scans hard drives and logs info
about every fine on the drive.

The first iteration of my code used a class and a generic list
to store the data and rhis took 13min on my 60 GB drive.

I wanted it to be quicker.

So the second time I changed the class to a struct and still
with the generic list got the time down to 4 min.

I am wondering if I can improve on this even more??

Anyone know if it would be possible to improve this
using C# or would it need to be wrritten in assembler or the like?

rotsey
 
R

Rotsey

Sorry Stephany!!!

I see well that explains 2 min approx for C Drive.

I am trying to gauge what would be the case
for my app on the typical setup either business or
domestic.

Thanks for that.


Stephany Young said:
The earlier run was on a super grunty server with really fast scsi drives,
truckloads of ram and multiple dual-core perocessors.


Rotsey said:
Oh i see you mean you ran my code on your machine.

surely it can't be that fast

I have a 7200RPM 60GB drive in 1.86GHz Pentium M Laptop

What is your specs?


Rotsey said:
Sorry Stephen what do you mean?



It was YOUR code!


Sorry about that , but it just a class with the generic list and a
string drive property

All I am doing is displaying some of the info when ScanFolder returns
But that is after the scan completes of course which is not the
problem.

Any chance of seeing your code???

public class DriveData

{

private List<PieFileInfo> mList = new List<PieFileInfo>();

public List<PieFileInfo> DataList

{

get { return mList; }

set { mList = value; }

}

private string mDrive;

public string Drive

{

get { return mDrive; }

set { mDrive = value; }

}

}

Approx. 13 seconds on a 250GB disk with 30,000+ directories and
1,000,000+ files.

What is DriveData? (You omitted to post it).

What do you do with the content of DriveData when the call to
ScanFolder(...) returns?



OK. Here is the class the scans the drive.

Can you see any issues here?

Bela, would you be able to post your code that scans the drive.

I am scanning 56GB in about 10 minutes.
public struct PieFileInfo

{

public int RowID;

public PieInfoType Type;

public string Name;

public string FullName;

public double Filesize;

public DateTime LastAccess;

public DateTime Created;

public string Extension;

public DateTime LastWrite;

public int ParentFolderRowID;

public string Drive;

public double FileAge1;

public double FileAge2;

public double FileAge3;

public bool IsRoot;

}

public class ScanDrive

{

private DriveData mList = new DriveData();

public DriveData DataList

{

get { return mList; }

set { mList = value; }

}

private string mDrive;

private int mNewFolderID = 0;

public ScanDrive(string drivefolder)

{

mDrive = drivefolder;

mList.Drive = drivefolder;

}

public bool ScanFolder(string folder, int folderID, ref double
psize1, ref double psize2, ref double psize3)

{

DirectoryInfo objDir = new DirectoryInfo(folder);

int newFolderID;

double size1 = 0;

double size2 = 0;

double size3 = 0;

System.DateTime fileDate;

try

{

newFolderID = WriteRow(objDir, folderID);

if (newFolderID == 0)

{

return false;

}

foreach (DirectoryInfo dir in objDir.GetDirectories())

{

if (!ScanFolder(dir.FullName, newFolderID, ref psize1,ref psize2,ref
psize3))

{

return true;

}

size1 += psize1;

size2 += psize2;

size3 += psize3;

}

foreach (FileInfo fle in objDir.GetFiles())

{

WriteRow(fle, newFolderID);

fileDate = GetAgeDate(fle);

if (fileDate >=
DateTime.Today.AddDays(Settings.Instance.SettingsData.Age1Days
* -1))

size1 += fle.Length;

else if (fileDate >=
DateTime.Today.AddDays(Settings.Instance.SettingsData.Age2Days
* -1))

size2 += fle.Length;

else

size3 += fle.Length;

}

UpdateFolderRow(newFolderID, size1, size2, size3);

psize1 = size1;

psize2 = size2;

psize3 = size3;

return true;

}

catch (Exception)

{

return true;

}



}

private DateTime GetAgeDate(FileInfo f)

{

if (f.CreationTime >=
DateTime.Today.AddDays(Settings.Instance.SettingsData.Age1Days
* -1))

return f.CreationTime;

else if (f.LastWriteTime >=
DateTime.Today.AddDays(Settings.Instance.SettingsData.Age2Days
* -1))

return f.LastWriteTime;

else

return f.LastAccessTime;

}

private int WriteRow(DirectoryInfo dir, int folderID)

{

PieFileInfo info = new PieFileInfo();

if (mNewFolderID == 0)

info.IsRoot = true;

mNewFolderID++;

info.RowID = mNewFolderID;

info.Type = PieInfoType.Folder;

info.FullName = dir.FullName;

info.Name = dir.Name;

info.LastAccess = dir.LastAccessTime;

info.LastWrite = dir.LastWriteTime;

info.Created = dir.CreationTime;

info.ParentFolderRowID = folderID;

info.Drive = mDrive;

mList.DataList.Add(info);

return mNewFolderID;

}

private int WriteRow(FileInfo file, int folderID)

{

PieFileInfo info = new PieFileInfo();

info.Type = PieInfoType.File;

info.Name = file.Name;

info.FullName = file.FullName;

info.Filesize = file.Length;

info.Extension = file.Extension;

info.LastAccess = file.LastAccessTime;

info.LastWrite = file.LastWriteTime;

info.Created = file.CreationTime;

info.ParentFolderRowID = folderID;

info.Drive = mDrive;

if (file.LastAccessTime >= DateTime.Today.AddDays(-30))

info.FileAge1 = file.Length;

else if (file.LastAccessTime >= DateTime.Today.AddDays(-60))

info.FileAge2 = file.Length;

else

info.FileAge3 = file.Length;

mList.DataList.Add(info);

return folderID;

}

private void UpdateFolderRow(int folderID, double size1, double
size2, double size3)

{

int index = mList.DataList.FindIndex(delegate(PieFileInfo fi)

{

if (fi.Type == PieInfoType.Folder && fi.RowID == folderID)

return true;

else

return false;

});

PieFileInfo info = mList.DataList[index];

info.FileAge1 += size1;

info.FileAge2 += size2;

info.FileAge3 += size3;

info.Filesize += size1 + size2 + size3;

mList.DataList[index] = info;

}

}



What do you scan? I do a directory and File scan in my 100GB (64GB
of data) hard drive and took 24 seconds. (Scanned every directory
and every file in the drive for info).

PD: for reference 170,187 files and 21,899 folders.


Regards,

Bela Istok

message Hi,

I am writing an app that scans hard drives and logs info
about every fine on the drive.

The first iteration of my code used a class and a generic list
to store the data and rhis took 13min on my 60 GB drive.

I wanted it to be quicker.

So the second time I changed the class to a struct and still
with the generic list got the time down to 4 min.

I am wondering if I can improve on this even more??

Anyone know if it would be possible to improve this
using C# or would it need to be wrritten in assembler or the like?

rotsey
 
B

Bela Istok

The first I see in you code is the use of Doubles, you don't need dobles you
only need long types for the size value.

You code run in 4 min. Going to see if can perform better.

Regards,

Bela Istok
Rotsey said:
OK. Here is the class the scans the drive.

Can you see any issues here?

Bela, would you be able to post your code that scans the drive.

I am scanning 56GB in about 10 minutes.
public struct PieFileInfo

{

public int RowID;

public PieInfoType Type;

public string Name;

public string FullName;

public double Filesize;

public DateTime LastAccess;

public DateTime Created;

public string Extension;

public DateTime LastWrite;

public int ParentFolderRowID;

public string Drive;

public double FileAge1;

public double FileAge2;

public double FileAge3;

public bool IsRoot;

}

public class ScanDrive

{

private DriveData mList = new DriveData();

public DriveData DataList

{

get { return mList; }

set { mList = value; }

}

private string mDrive;

private int mNewFolderID = 0;

public ScanDrive(string drivefolder)

{

mDrive = drivefolder;

mList.Drive = drivefolder;

}

public bool ScanFolder(string folder, int folderID, ref double psize1, ref
double psize2, ref double psize3)

{

DirectoryInfo objDir = new DirectoryInfo(folder);

int newFolderID;

double size1 = 0;

double size2 = 0;

double size3 = 0;

System.DateTime fileDate;

try

{

newFolderID = WriteRow(objDir, folderID);

if (newFolderID == 0)

{

return false;

}

foreach (DirectoryInfo dir in objDir.GetDirectories())

{

if (!ScanFolder(dir.FullName, newFolderID, ref psize1,ref psize2,ref
psize3))

{

return true;

}

size1 += psize1;

size2 += psize2;

size3 += psize3;

}

foreach (FileInfo fle in objDir.GetFiles())

{

WriteRow(fle, newFolderID);

fileDate = GetAgeDate(fle);

if (fileDate >=
DateTime.Today.AddDays(Settings.Instance.SettingsData.Age1Days * -1))

size1 += fle.Length;

else if (fileDate >=
DateTime.Today.AddDays(Settings.Instance.SettingsData.Age2Days * -1))

size2 += fle.Length;

else

size3 += fle.Length;

}

UpdateFolderRow(newFolderID, size1, size2, size3);

psize1 = size1;

psize2 = size2;

psize3 = size3;

return true;

}

catch (Exception)

{

return true;

}



}

private DateTime GetAgeDate(FileInfo f)

{

if (f.CreationTime >=
DateTime.Today.AddDays(Settings.Instance.SettingsData.Age1Days * -1))

return f.CreationTime;

else if (f.LastWriteTime >=
DateTime.Today.AddDays(Settings.Instance.SettingsData.Age2Days * -1))

return f.LastWriteTime;

else

return f.LastAccessTime;

}

private int WriteRow(DirectoryInfo dir, int folderID)

{

PieFileInfo info = new PieFileInfo();

if (mNewFolderID == 0)

info.IsRoot = true;

mNewFolderID++;

info.RowID = mNewFolderID;

info.Type = PieInfoType.Folder;

info.FullName = dir.FullName;

info.Name = dir.Name;

info.LastAccess = dir.LastAccessTime;

info.LastWrite = dir.LastWriteTime;

info.Created = dir.CreationTime;

info.ParentFolderRowID = folderID;

info.Drive = mDrive;

mList.DataList.Add(info);

return mNewFolderID;

}

private int WriteRow(FileInfo file, int folderID)

{

PieFileInfo info = new PieFileInfo();

info.Type = PieInfoType.File;

info.Name = file.Name;

info.FullName = file.FullName;

info.Filesize = file.Length;

info.Extension = file.Extension;

info.LastAccess = file.LastAccessTime;

info.LastWrite = file.LastWriteTime;

info.Created = file.CreationTime;

info.ParentFolderRowID = folderID;

info.Drive = mDrive;

if (file.LastAccessTime >= DateTime.Today.AddDays(-30))

info.FileAge1 = file.Length;

else if (file.LastAccessTime >= DateTime.Today.AddDays(-60))

info.FileAge2 = file.Length;

else

info.FileAge3 = file.Length;

mList.DataList.Add(info);

return folderID;

}

private void UpdateFolderRow(int folderID, double size1, double size2,
double size3)

{

int index = mList.DataList.FindIndex(delegate(PieFileInfo fi)

{

if (fi.Type == PieInfoType.Folder && fi.RowID == folderID)

return true;

else

return false;

});

PieFileInfo info = mList.DataList[index];

info.FileAge1 += size1;

info.FileAge2 += size2;

info.FileAge3 += size3;

info.Filesize += size1 + size2 + size3;

mList.DataList[index] = info;

}

}



Bela Istok said:
What do you scan? I do a directory and File scan in my 100GB (64GB of
data) hard drive and took 24 seconds. (Scanned every directory and every
file in the drive for info).

PD: for reference 170,187 files and 21,899 folders.


Regards,

Bela Istok
 
L

Liz

Where is PieInfoType defined? do you have a driver stub which launches this
thing?


Rotsey said:
OK. Here is the class the scans the drive.

Can you see any issues here?

Bela, would you be able to post your code that scans the drive.

I am scanning 56GB in about 10 minutes.
public struct PieFileInfo

{

public int RowID;

public PieInfoType Type;

public string Name;

public string FullName;

public double Filesize;

public DateTime LastAccess;

public DateTime Created;

public string Extension;

public DateTime LastWrite;

public int ParentFolderRowID;

public string Drive;

public double FileAge1;

public double FileAge2;

public double FileAge3;

public bool IsRoot;

}

public class ScanDrive

{

private DriveData mList = new DriveData();

public DriveData DataList

{

get { return mList; }

set { mList = value; }

}

private string mDrive;

private int mNewFolderID = 0;

public ScanDrive(string drivefolder)

{

mDrive = drivefolder;

mList.Drive = drivefolder;

}

public bool ScanFolder(string folder, int folderID, ref double psize1, ref
double psize2, ref double psize3)

{

DirectoryInfo objDir = new DirectoryInfo(folder);

int newFolderID;

double size1 = 0;

double size2 = 0;

double size3 = 0;

System.DateTime fileDate;

try

{

newFolderID = WriteRow(objDir, folderID);

if (newFolderID == 0)

{

return false;

}

foreach (DirectoryInfo dir in objDir.GetDirectories())

{

if (!ScanFolder(dir.FullName, newFolderID, ref psize1,ref psize2,ref
psize3))

{

return true;

}

size1 += psize1;

size2 += psize2;

size3 += psize3;

}

foreach (FileInfo fle in objDir.GetFiles())

{

WriteRow(fle, newFolderID);

fileDate = GetAgeDate(fle);

if (fileDate >=
DateTime.Today.AddDays(Settings.Instance.SettingsData.Age1Days * -1))

size1 += fle.Length;

else if (fileDate >=
DateTime.Today.AddDays(Settings.Instance.SettingsData.Age2Days * -1))

size2 += fle.Length;

else

size3 += fle.Length;

}

UpdateFolderRow(newFolderID, size1, size2, size3);

psize1 = size1;

psize2 = size2;

psize3 = size3;

return true;

}

catch (Exception)

{

return true;

}



}

private DateTime GetAgeDate(FileInfo f)

{

if (f.CreationTime >=
DateTime.Today.AddDays(Settings.Instance.SettingsData.Age1Days * -1))

return f.CreationTime;

else if (f.LastWriteTime >=
DateTime.Today.AddDays(Settings.Instance.SettingsData.Age2Days * -1))

return f.LastWriteTime;

else

return f.LastAccessTime;

}

private int WriteRow(DirectoryInfo dir, int folderID)

{

PieFileInfo info = new PieFileInfo();

if (mNewFolderID == 0)

info.IsRoot = true;

mNewFolderID++;

info.RowID = mNewFolderID;

info.Type = PieInfoType.Folder;

info.FullName = dir.FullName;

info.Name = dir.Name;

info.LastAccess = dir.LastAccessTime;

info.LastWrite = dir.LastWriteTime;

info.Created = dir.CreationTime;

info.ParentFolderRowID = folderID;

info.Drive = mDrive;

mList.DataList.Add(info);

return mNewFolderID;

}

private int WriteRow(FileInfo file, int folderID)

{

PieFileInfo info = new PieFileInfo();

info.Type = PieInfoType.File;

info.Name = file.Name;

info.FullName = file.FullName;

info.Filesize = file.Length;

info.Extension = file.Extension;

info.LastAccess = file.LastAccessTime;

info.LastWrite = file.LastWriteTime;

info.Created = file.CreationTime;

info.ParentFolderRowID = folderID;

info.Drive = mDrive;

if (file.LastAccessTime >= DateTime.Today.AddDays(-30))

info.FileAge1 = file.Length;

else if (file.LastAccessTime >= DateTime.Today.AddDays(-60))

info.FileAge2 = file.Length;

else

info.FileAge3 = file.Length;

mList.DataList.Add(info);

return folderID;

}

private void UpdateFolderRow(int folderID, double size1, double size2,
double size3)

{

int index = mList.DataList.FindIndex(delegate(PieFileInfo fi)

{

if (fi.Type == PieInfoType.Folder && fi.RowID == folderID)

return true;

else

return false;

});

PieFileInfo info = mList.DataList[index];

info.FileAge1 += size1;

info.FileAge2 += size2;

info.FileAge3 += size3;

info.Filesize += size1 + size2 + size3;

mList.DataList[index] = info;

}

}



Bela Istok said:
What do you scan? I do a directory and File scan in my 100GB (64GB of
data) hard drive and took 24 seconds. (Scanned every directory and every
file in the drive for info).

PD: for reference 170,187 files and 21,899 folders.


Regards,

Bela Istok
 
B

Bela Istok

public enum PieInfoType
{
Folder,
File
}

Regards,

Bela Istok
Liz said:
Where is PieInfoType defined? do you have a driver stub which launches
this thing?


Rotsey said:
OK. Here is the class the scans the drive.

Can you see any issues here?

Bela, would you be able to post your code that scans the drive.

I am scanning 56GB in about 10 minutes.
public struct PieFileInfo

{

public int RowID;

public PieInfoType Type;

public string Name;

public string FullName;

public double Filesize;

public DateTime LastAccess;

public DateTime Created;

public string Extension;

public DateTime LastWrite;

public int ParentFolderRowID;

public string Drive;

public double FileAge1;

public double FileAge2;

public double FileAge3;

public bool IsRoot;

}

public class ScanDrive

{

private DriveData mList = new DriveData();

public DriveData DataList

{

get { return mList; }

set { mList = value; }

}

private string mDrive;

private int mNewFolderID = 0;

public ScanDrive(string drivefolder)

{

mDrive = drivefolder;

mList.Drive = drivefolder;

}

public bool ScanFolder(string folder, int folderID, ref double psize1,
ref double psize2, ref double psize3)

{

DirectoryInfo objDir = new DirectoryInfo(folder);

int newFolderID;

double size1 = 0;

double size2 = 0;

double size3 = 0;

System.DateTime fileDate;

try

{

newFolderID = WriteRow(objDir, folderID);

if (newFolderID == 0)

{

return false;

}

foreach (DirectoryInfo dir in objDir.GetDirectories())

{

if (!ScanFolder(dir.FullName, newFolderID, ref psize1,ref psize2,ref
psize3))

{

return true;

}

size1 += psize1;

size2 += psize2;

size3 += psize3;

}

foreach (FileInfo fle in objDir.GetFiles())

{

WriteRow(fle, newFolderID);

fileDate = GetAgeDate(fle);

if (fileDate >=
DateTime.Today.AddDays(Settings.Instance.SettingsData.Age1Days * -1))

size1 += fle.Length;

else if (fileDate >=
DateTime.Today.AddDays(Settings.Instance.SettingsData.Age2Days * -1))

size2 += fle.Length;

else

size3 += fle.Length;

}

UpdateFolderRow(newFolderID, size1, size2, size3);

psize1 = size1;

psize2 = size2;

psize3 = size3;

return true;

}

catch (Exception)

{

return true;

}



}

private DateTime GetAgeDate(FileInfo f)

{

if (f.CreationTime >=
DateTime.Today.AddDays(Settings.Instance.SettingsData.Age1Days * -1))

return f.CreationTime;

else if (f.LastWriteTime >=
DateTime.Today.AddDays(Settings.Instance.SettingsData.Age2Days * -1))

return f.LastWriteTime;

else

return f.LastAccessTime;

}

private int WriteRow(DirectoryInfo dir, int folderID)

{

PieFileInfo info = new PieFileInfo();

if (mNewFolderID == 0)

info.IsRoot = true;

mNewFolderID++;

info.RowID = mNewFolderID;

info.Type = PieInfoType.Folder;

info.FullName = dir.FullName;

info.Name = dir.Name;

info.LastAccess = dir.LastAccessTime;

info.LastWrite = dir.LastWriteTime;

info.Created = dir.CreationTime;

info.ParentFolderRowID = folderID;

info.Drive = mDrive;

mList.DataList.Add(info);

return mNewFolderID;

}

private int WriteRow(FileInfo file, int folderID)

{

PieFileInfo info = new PieFileInfo();

info.Type = PieInfoType.File;

info.Name = file.Name;

info.FullName = file.FullName;

info.Filesize = file.Length;

info.Extension = file.Extension;

info.LastAccess = file.LastAccessTime;

info.LastWrite = file.LastWriteTime;

info.Created = file.CreationTime;

info.ParentFolderRowID = folderID;

info.Drive = mDrive;

if (file.LastAccessTime >= DateTime.Today.AddDays(-30))

info.FileAge1 = file.Length;

else if (file.LastAccessTime >= DateTime.Today.AddDays(-60))

info.FileAge2 = file.Length;

else

info.FileAge3 = file.Length;

mList.DataList.Add(info);

return folderID;

}

private void UpdateFolderRow(int folderID, double size1, double size2,
double size3)

{

int index = mList.DataList.FindIndex(delegate(PieFileInfo fi)

{

if (fi.Type == PieInfoType.Folder && fi.RowID == folderID)

return true;

else

return false;

});

PieFileInfo info = mList.DataList[index];

info.FileAge1 += size1;

info.FileAge2 += size2;

info.FileAge3 += size3;

info.Filesize += size1 + size2 + size3;

mList.DataList[index] = info;

}

}



Bela Istok said:
What do you scan? I do a directory and File scan in my 100GB (64GB of
data) hard drive and took 24 seconds. (Scanned every directory and every
file in the drive for info).

PD: for reference 170,187 files and 21,899 folders.


Regards,

Bela Istok

Hi,

I am writing an app that scans hard drives and logs info
about every fine on the drive.

The first iteration of my code used a class and a generic list
to store the data and rhis took 13min on my 60 GB drive.

I wanted it to be quicker.

So the second time I changed the class to a struct and still
with the generic list got the time down to 4 min.

I am wondering if I can improve on this even more??

Anyone know if it would be possible to improve this
using C# or would it need to be wrritten in assembler or the like?

rotsey
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Top