Need a fast way(c#) to get directory of files over a WAN using selection criteria

  • Thread starter Thread starter ganderso
  • Start date Start date
G

ganderso

Help!!

I need to interrogate about 150,000 files across a WAN and pull in ONLY
the files less than 5 days old and larger than 10k bytes. I'm currently
using Directory.GetFiles and it's taking hours!!

Anyone have any great ideas? Any help will be greatly appreciated.

Thanks
Greg
 
Another way would be to create a small console app on the server side that
does the search and outputs a txt file with the filenames that match your
search. Launch this program using psexec or something from the client and
wait for it to finish then copy that file local and parse the file names to
string[]. Then download each of those files. That should run much faster.
 
Relevant part of code:

minimumFilesize =
ConfigurationSettings.AppSettings["minimum_filesize"];
numberDays = ConfigurationSettings.AppSettings["number_days"];
try
{
DisplayMsg("Getting Directory info for... " + sDir);
foreach (string f in Directory.GetFiles(sDir, "*.tif"))
{
txtViewed.Text = Convert.ToString(Convert.ToInt32(txtViewed.Text)
+ 1);
FileInfo fi = new FileInfo(f);
//textBox1.Text = ("File size of mscdrun.cab: {0}" +
fi.Length.ToString());

if (fi.CreationTime >
DateTime.Now.AddDays(-Convert.ToInt32(numberDays)))
{
if (fi.Length > Convert.ToInt32(minimumFilesize))
{
 
Unfortunately - I have only read access to these remote servers.

I like the idea if I could just put a little program out there to run
time-scheduled and create a nice little text file of all the qualifying
Tif files. If I can't get anything else to work this may have to be
done.

Thanks
Greg
 
Relevant part of code:

minimumFilesize =
ConfigurationSettings.AppSettings["minimum_filesize"];
numberDays = ConfigurationSettings.AppSettings["number_days"];
try
{
DisplayMsg("Getting Directory info for... " + sDir);
foreach (string f in Directory.GetFiles(sDir, "*.tif"))
{
txtViewed.Text = Convert.ToString(Convert.ToInt32(txtViewed.Text)
+ 1);
FileInfo fi = new FileInfo(f);
//textBox1.Text = ("File size of mscdrun.cab: {0}" +
fi.Length.ToString());

if (fi.CreationTime >
DateTime.Now.AddDays(-Convert.ToInt32(numberDays)))
{
if (fi.Length > Convert.ToInt32(minimumFilesize))
{
Click to expand...

I would pre-calculate some of those values, instead of doing it
over and over again:
- store the cutoff-date as DateTime (once)
- pre-convert mimimumFilesize to int (once)
- use a local int to store number of files read, and convert *that* to string for the textbox
or, if you know how many files to check, use a progressbar instead of a textbox,
might be more efficient (writing to screen is expensive)

This might shave off some time, but I doubt it will gain you "hours".

Hans Kesting
 
Unfortunately - I have only read access to these remote servers.

I like the idea if I could just put a little program out there to run
time-scheduled and create a nice little text file of all the qualifying
Tif files. If I can't get anything else to work this may have to be
done.

Thanks
Greg
Click to expand...

If those boxes are open for WMI queries, you could use System.Management and
WMI's class "cim_datafile"

for instance following query returns all files that have a size > 5000000
bytes from d:\developertools that have been modified since June 5th, 2005

select * from cim_datafile where drive='d:' and path='\\developertools\\'
and filesize > 5000000
and lastmodified > ""20050605000000.000000+000"

When you connect to the remote box, the query is executed by the remote WMI
service and the result is send back to the client when done, running such
query on a remote box is just as fast as on a local box, for instance it
only takes less than 5 seconds to query a complete 40GB disk using next
query.
select * from cim_datafile where drive='d:' and filesize > 5000000
and lastmodified > ""20010605000000.000000+000"
Willy.
 
Back
Top