Multiple Files Problem with FileSystemWatcher

[Guest]

Hello,

I am building a data management application with the following processes:

Process 1 is a Windows service which uses FileSystemWatcher to monitor a
directory.
Process 2 opens a file copied into the directory and inserts the data into a
data warehouse.
Process 3 queries the data warehouse and transfers the results to a data mart.

Currently, Processes 2 and 3 run from Process 1 but this might not be
optimal. Here is the code:

private void FileMonitor_Changed(object sender,
System.IO.FileSystemEventArgs e)
{
string ChangeType = e.ChangeType.ToString();
string renamed = e.FullPath;

if (ChangeType=="Created")
{
//Declare and instantiate Process 2.
System.Diagnostics.Process process2;
process2= new System.Diagnostics.Process();
process2.EnableRaisingEvents = false;
string strCmdLine;

//Process 2 opens file and inserts data into a Data Warehouse
strCmdLine2 = "/C C:\\MYFILES\\Programs\\LoadToWarehouse.exe \"" + renamed
+ "\"";

// Wait for the file to finish copying.
// Is there a more robust way than Sleep(1000)
// to know when copying has completed?
Thread.Sleep(1000);

process2.StartInfo.FileName = "CMD.exe";
process2.StartInfo.Arguments = strCmdLine2;
process2.StartInfo.UseShellExecute = false;
process2.StartInfo.RedirectStandardOutput = true;
process2.StartInfo.RedirectStandardError = true;
process2.StartInfo.WindowStyle = ProcessWindowStyle.Hidden;
process2.StartInfo.CreateNoWindow = true;
process2.Start();
StreamReader myStreamReader = process2.StandardOutput;
StreamReader myStreamError = process2.StandardError;

// rdfID is needed for Process 3
string rdfID = myStreamReader.ReadLine();
string stdError = myStreamError.ReadLine();
process2.Close();
EventLog.WriteEntry("Directory Monitor", "stdError: " + stdError);


//Declare and instantiate Process 3.
System.Diagnostics.Process process3;
process3= new System.Diagnostics.Process();
process3.EnableRaisingEvents = false;
string strCmdLine2;

// Process 3 queries the Data Warehouse against rdfID
// and inserts the results into a Data Mart
strCmdLine3 = "/C C:\\MYFILES\\Programs\\TransferToDataMart.exe \"" +
rdfID + "\"";

process3.StartInfo.WorkingDirectory = "C:\\Temp";
process3.StartInfo.FileName = "CMD.exe";
process3.StartInfo.Arguments = strCmdLine3;
process3.StartInfo.UseShellExecute = false;
process3.StartInfo.RedirectStandardError = true;
process3.StartInfo.WindowStyle = ProcessWindowStyle.Hidden;
process3.StartInfo.CreateNoWindow = true;
process3.Start();

StreamReader myStreamError2 = process3.StandardError;
string stdError2 = myStreamError2.ReadLine();
EventLog.WriteEntry("Directory Monitor", "stdError2: " + stdError2);
process3.Close();


}

}

This application works as expected if only one file is copied to the
directory. If multiple files are copied then things go out of sync. Process 3
is expensive and could take more than a few seconds. In this case additional
files would have been copied to the directory undetected by Process 1 (the
FileSystemWatcher process) because it still wouldn’t have completed.

I’m still fairly new to .Net and I’m assuming that processing multiple files
copied to a single directory monitored by FileSystemWatcher is commonplace.
Can you experts please suggest an approach that would overcome my
synchronisation problem? It would probably help if there was a way in which
Process 1 would know for certain that a file has completed copying other than
using Thread.sleep with a large value.

Thanks in advance and have a great day!

 

[Guest]

Since it sounds like your requirements don't indicate that Process 1 needs a
response from Process 2 when it is complete. I would pull those pieces of
functionality out into their own method and call it asynchronously.

The net affect of this will be that your file watcher will just simply watch
the for the files and start the execution off for Process 2 and 3. You will
then likely end up in a scenario where you might have multiple Process 2 and
3 processes running side by side for different files, so you might want to be
careful that they data being inserted side by side would not cause any
problems, else, you will need to think about syncronization requirements.

Hope that helps,
Patrick Altman