FileSystemWatcher limits problem

[Phil396]

I have a windows service that uses a filesystemwatcher
to wait for files and process them to a database.
Sometimes a large group of files will be cut and paste
for the filesystemwatcher to process. I ran into
trouble when trying to process a group of files
at a time.
The code below fixed my problem for about 15 large files
( around 100k each ). However when I tried to use it
on a group of small files ( 90 files ) it crashes (
around 6k each ). I am going to try increasing the time
for thread sleep. Any other solutions ?
Here is my code for the file system watcher using a
windows service to wait for files-

private void ProcessFiles(string FullPath)
{
while (true)
{
try
{
objFiles.Parse(FullPath);
break;
}
catch (System.IO.IOException)
{
System.Threading.Thread.Sleep( 1000 );
}
}

 

[Dan Bass]

This isn't realllly what the FileSystemWatcher is for. It's used more in the
context of logging file changes on system, rather than actually reading out
the file contents. I've heard of previous issues on this where the files
"exist" at the designated location, but the copy has not been completed.
This then leads to access violations on shared files etc etc...

What I've done in the past is to (for synchronous solutions) write my own
class that is responsible for polling a given folder. When new files arrive,
the array of file names are then used to extract the data, and pass it back
to the object creator. This allows you to then put in exception handling for
different errors you could get on an NT file system. One example is that
there was a real time anti virus process running in the background, and if
this happened to "monitor" a file while I was trying to access and remove
the file, then an exception occurs. This knowledge means I can plan for it
my own way...

I've included a snippet of some work I did a while back...

// some data members (set with an initialisation method)
protected string pathToMonitor = "";
protected string fileSearchPattern = "*.xml";
protected Queue foundFiles = new Queue();
protected string fileToClear = "";


// ------------------------------------------------------------------------
// Receive Methods
// ------------------------------------------------------------------------

// call this to sit and wait for data, returns
OperationResult.Success
// with data containing one files contents.
public override OperationResult Receive( out string data )
{
data = "";
dataBufferToClear = "";

/*** Status.Listening is a static turned on and off by the start
up/ shut down methods of the windows service ***/
while ( Status.Listening )
{

//
// the queueing system, although a little more complex,
// ensures that the files are processed in some order
// according to when they arrived.
//

if ( foundFiles.Count == 0 )
{
// get all the files matching our criteria
string[] Files = Directory.GetFiles( pathToMonitor,
fileSearchPattern );

if ( Files != null )
{
foreach ( string fileName in Files )
{
foundFiles.Enqueue ( fileName );
}
}

/*** a method that checks Status.Listening stays true
while it waits for 5 seconds ***/
Methods.SleepItOff( 5 );

}


// if the queue is not empty, then there's new files
if ( foundFiles.Count == 0 )
{
continue;
}

string currentFilename = (string)foundFiles.Dequeue();

/*** GetFileContents simply opens the file, extracts the
data and closes the reader again ***/
data = Methods.GetFileContents(currentFilename);


//
// finally set the file for removal

fileToClear = currentFilename;

if ( data != null )
{
if ( data != "" )
{
dataBufferToClear = data;

/*** my own result enumerations ***/
return OperationResult.Success ;
}
}

Methods.SleepItOff(5);

}

return OperationResult.NoMessageFound;

}

// call this when you've successfully put the file into the database
// If an error occurs in the processing of a file, then you know
// that the file data is still there and hasn't been lost.
public override void CompleteReceive()
{
string data = Methods.GetFileContents(fileToClear);

if ( dataBufferToClear == "" || dataBufferToClear == data )
{
Methods.DeleteFile ( fileToClear );
}

}

Hope this helps.

Dan.