FileSystemWatcher and open files

[David Jackson]

Hello,

I have a Windows service written in C# which uses FileSystemWatcher to
monitor a network folder for the creation of certain files. These are text
files which originate from a remote mainframe system.

When (a) file(s) arrive in the network folder, the Created event fires which
then kicks of another method within the Windows service which processes the
file.

However, the files might be several megabytes in length, and the problem I'm
having is that the FileSystemWatcher event fires as soon as the file
creation process begins, not when it ends, so for larger files this can fail
because the file itself is still being written when the Windows service
tries to start processing it.

I understand that this is by design.

Having done a bit of a trawl through Google, I found this article:
http://www.thescripts.com/forum/thread447494.html which suggests (among
other things), to set up a loop with a sleep timer to keep trying to rename
the file, the idea being that as soon as it is possible to rename the file
that must mean that the creation process has finished. Makes sense.

I also found this Microsoft article:
http://www.thescripts.com/forum/thread447494.html which suggests just
putting the thread to sleep for 500ms anyway, thereby giving the file
creation process enough time to complete. I could easily set that to
something huge like 30 seconds, I suppose, but that doesn't seem
particularly robust or efficient.

I'd be interested how other people have solved this issue.

Thank you.

DJ

 

[Family Tree Mike]

Hello,

I have a Windows service written in C# which uses FileSystemWatcher to
monitor a network folder for the creation of certain files. These are text
files which originate from a remote mainframe system.

When (a) file(s) arrive in the network folder, the Created event fires which
then kicks of another method within the Windows service which processes the
file.

However, the files might be several megabytes in length, and the problem I'm
having is that the FileSystemWatcher event fires as soon as the file
creation process begins, not when it ends, so for larger files this can fail
because the file itself is still being written when the Windows service
tries to start processing it.

I understand that this is by design.

Having done a bit of a trawl through Google, I found this article:
http://www.thescripts.com/forum/thread447494.html which suggests (among
other things), to set up a loop with a sleep timer to keep trying to rename
the file, the idea being that as soon as it is possible to rename the file
that must mean that the creation process has finished. Makes sense.

I also found this Microsoft article:
http://www.thescripts.com/forum/thread447494.html which suggests just
putting the thread to sleep for 500ms anyway, thereby giving the file
creation process enough time to complete. I could easily set that to
something huge like 30 seconds, I suppose, but that doesn't seem
particularly robust or efficient.

I'd be interested how other people have solved this issue.

Thank you.

DJ

It depends on how much latency your situation can accept between arival of
the file and processing to begin. For example, if the files represent posts
of some type of data that is aggregated with other posts hourly, then a 30
sec-1 minute delay has no effect on the end requirement. If it is a failure
notification, then the processing may be more urgent.

 

[Dag Christensen]

We're running a windows service which contains several filesystemwatchers.
Sometimes we hit spikes of 20-50.000 files in a matter of seconds and each
file can potentially take some time to process. There's two problems we had
to solve because of this:

1. If you spend too long time processing each event, and events arrive
quickly, the internal Win32 filesystemwatcher buffer will fill up and start
dropping events. See the InternalBufferSize property in FileSystemWatcher
docs for more information

2. Files not being closed when the event is raised (as you describe below)

Solution for #1 was to spend as little time as possible in the event
handler. In our solution we simply post the data to a queue and let a
separate worker thread process it.

In the worker thread we check each file before processing them. Some files
must be handled in the order they arrive, so the queue will stall until the
file at the head of the queue is closed.

Other files can be handled in any order, and we tag them with the worker
thread iteration id and push them back to the end of the queue (to prevent
endless processing of a opened file in a tight loop, we use the iteration id
to check if we should Thread.Sleep or not before processing the next item in
the queue).

To check if a file is available/closed, we use the following code which
seems to be fast and robust enough. It's a greedy catch but I'm not changing
critial code that works

bool IsFileClosed( FileInfo fi )
{
try
{
FileStream fs = fi.Open( FileMode.Open, FileAccess.ReadWrite,
FileShare.None );
fs.Close();
fs.Dispose();
return true;
}
catch
{
return false;
}
}

 

[David Jackson]

Thanks for the reply.

To check if a file is available/closed, we use the following code which
seems to be fast and robust enough. It's a greedy catch but I'm not
changing critial code that works

bool IsFileClosed( FileInfo fi )
{
try
{
FileStream fs = fi.Open( FileMode.Open, FileAccess.ReadWrite,
FileShare.None );
fs.Close();
fs.Dispose();
return true;
}
catch
{
return false;
}
}

Presumably you are calling the above function in some sort of a timer-based
loop which is waiting until the function returns true?

DJ

 

[Dag Christensen]

Yes, that's right.

A very simple implementation of the worker thread looks something like this.
You'll at least need extra logic for thread synchronization and error
handling.

while ( !shutdownRequested )
{
if (queue.Count != 0)
{
WorkItem item = queue.Peek();
if (IsFileClosed(item.File))
{
queue.Dequeue();
ProcessWorkItem (item);
// no delay since queue might contain more items
}
else
{
Thread.Sleep(200); // file is busy, short delay before checking
again
}
}
else
{
Thread.Sleep(1000); // while the queue is empty
}
}