Using FileSystemWatcher with large files

[Josh Usovsky]

I'm setting up a watched folder using FileSystemWatcher.
When I drop a small file into the watched folder, I can
respond to a .Created event and process the file with
other code. However, if I try copying a large file into
the watched folder, the .Created event is fired off
immediately and not when the file is finished copying. I
have tried waiting for a certain number of .Changed
events to be fired after the initial .Created event, and
that method works greate except for the fact that some
files trigger 3 .Changed events, and others 5, 6, and 7.
I can't find a pattern to determine why there are a
differing number of events being fired for the various
files. Does anyone else have experience in this or have
any suggestions for determining when each file is
finished being written ? I'd rather not use a checking
loop or error handling for program flow, but I might have
to at this point.

Thanks in advance,
--JU

 

[Martin Robins]

Josh,

Whenever I use a FileSystemWatcher in this way I process the created files
in new threads. Each thread is added to a hashtable that is referenced by
the file name thus allowing me to see if i have already created a processor
for the file whe subsequent events are fired.

Within the processor thread, I open the file exclusively and continuously
loop until the open is sucessful.

Martin.

 

[David Sworder]

I've had similar problems using FileSystemWatcher to process large files
which are FTP'd to us each night. The event firing mechanism in
FileSystemWatcher is just flakey. While not ideal, using a polling loop is
the method that I use and it seems to be the only way around this problem.
When FileSystemWatcher fires the event, immediately try to open the file in
exclusive mode. If an exception is thrown, set the timer and then try to
open the file again when the timer fires. Once the file is available, stop
the timer and process the file.

A simpler method is to respond to the FileSystemWatcher event by
entering a loop which includes a Thread.Sleep() call after each failure to
open the file. The downside of this approach is that you're causing a
threadpool thread to block during the Sleep() call. Nevertheless, 95% of the
time the Sleep() loop is the way to go. It's just simpler.

David

 

[Mark]

The event firing mechanism in FileSystemWatcher is just
flakey.

The event firing system is Windows, and it's not
guaranteed to be one event per file operation.

If you read the documentation, you'll find where it says
"Common file system operations might raise more than one
event" and "when a file is moved from one directory to
another, several OnChanged and some OnCreated and OnDeleted
events might be raised".

Mark

 

[David Sworder]

If you read the documentation, you'll find where it says

"Common file system operations **might** raise more than one
event" and "when a file is moved from one directory to
another, **several** OnChanged and some OnCreated and OnDeleted
events **might** be raised".

It's words like "might" and "several" that bother me. I wish things were more precise. Not to say that FileSystemWatcher is useless by any means -- it's just not as consistent as I'd prefer.

David

 

[Mark]

It's words like "might" and "several" that bother me. I wish
things were more precise. Not to say that FileSystemWatcher is
useless by any means -- it's just not as consistent as I'd prefer.

Agreed, it'd be nice if things were more concrete. Just don't "blame the
messenger", like they say. The FSW here is just a messenger. Things happen
this way because of the way filesystem access works in windows.

Mark