FileSystemWatcher and large files

D

David Jackson

Hello,

I'm presently writing a Windows service which waits for files to be
downloaded from various completely separate remote systems into a specific
network folder (the service itself doesn't do the downloads) and then
processes them and consolidates them into the company database.

Therefore I thought that FileSystemWatcher would be a good solution for
this.

However, I soon noticed that the Created event fires as soon as a file
begins to be written, and that the FileSystemWatcher class doesn't appear to
have any method or event to say that the creation process has completed. As
some of these files can be quite large (several GB), the write process can
take several minutes. This initially caused me some problems because the
service tried to begin processing the file before it had been completely
written.

After doing some research on Google, MSDN and other places, there seemed to
be two generally accepted solutions for this problem.

1) As soon as the Created event fires, make the thread sleep for, say 30
seconds to allow the file to be completely written

2) Set up a loop to keep checking if it's possible to open the file
exclusively, e.g.

bool IsFileClosed(FileInfo fi)
{
bool isFileClosed = false;
try
{
using (FileStream fs = fi.Open( FileMode.Open, FileAccess.ReadWrite,
FileShare.None))
{
fs.Close();
return true;
}
}
catch
{
return false;
}
}


I appreciate that the second option is probably more reliable because there
is no guarantee that the number of seconds that the thread is put to sleep
will be sufficient for the file to be written, but I'd be grateful to know
if anyone has a better solution, or any comments on the above.

Thank you.

DJ
 
P

Peter Morris

I went with option 2. Try each file in the list, if I can a lock conflict
then skip it. If I get to the end of the list and there are still files in
the list I sleep for X seconds, if I get to the end of the list and there
are no items in it then I Monitor.Wait() for a new one to be added.
 
D

David Jackson

I went with option 2. Try each file in the list, if I can a lock conflict
then skip it. If I get to the end of the list and there are still files
in the list I sleep for X seconds, if I get to the end of the list and
there are no items in it then I Monitor.Wait() for a new one to be added.

Sounds good, thank you.
 
W

William Stacey

If possible, I would write a "trigger" file (maybe zero length) as the last
file pushed. Then you can just wait on that file to be created and you know
the others are done and closed. In the same way the pusher can write an
error file.
 
D

David Jackson

If possible, I would write a "trigger" file (maybe zero length) as the
last file pushed. Then you can just wait on that file to be created and
you know the others are done and closed. In the same way the pusher can
write an error file.

Thanks for the reply.

As mentioned, the Windows service doesn't actually do the downloads.

Also, the files aren't downloaded in a discreet batch - any number of them
can appear at any time through the day.

DJ
 
D

David Jackson

I went with option 2. Try each file in the list, if I can a lock conflict
then skip it. If I get to the end of the list and there are still files
in the list I sleep for X seconds, if I get to the end of the list and
there are no items in it then I Monitor.Wait() for a new one to be added.

Is it possible for only one app to monitor a given folder at any given time?
This appears to be the case as far as I can see.

E.g. if app A sets up a monitor on a network folder, but then app B sets up
a monitor on the same folder, app A no longer receives notifications.

I can understand why this might be the case, but can anyone confirm this?

Thank you.

DJ
 
D

David Jackson

Thanks for the reply.
I wouldn't have thought so.

Looks like you're correct.
Do you only have problems when trying to monitor a network folder, or can
you reproduce the issue with a local volume?

Seens to be any folder on any volume.
It would be easier to confirm the issue if you'd post a
concise-but-complete code sample that reliably demonstrates the issue. :)

1) Create an app (appA) which sets up a FileSystemWatcher on a folder to
respond to new files being created in that folder.

2) Create another app (appB) which sets up a FileSystemWatcher on a folder
to respond to new files being created in the same folder.

3) Run appA.

4) Create a new file in the folder above - note that appA receives
notification of the newly created file.

5) Run appB.

6) Create a new file in the folder above - note that appA doesn't receive
notification of the newly created file but appB does.
 
D

David Jackson

Thanks again for the reply.
If it looks to as though I'm correct, then you are not having the problem
you're saying you are having. :)

I'm not really having a problem as such, just seeking clarification of an
observation.
I probably should have been more careful with my phrasing.

Me too, seemingly :)

Thanks very much for the links.
If multiple FileSystemWatcher objects are watching the same UNC path
in Windows XP prior to Service Pack 1, or Windows 2000 SP2 or earlier,
then only one of the objects will raise an event. On machines running
Windows XP SP1 and newer, Windows 2000 SP3 or newer or Windows Server
2003, all FileSystemWatcher objects will raise the appropriate events.

Maybe your OS just isn't up-to-date?

I'm using 64-bit Vista Business edition, and the folder in question is on an
NTFS partition in a machine running 64-bit Windows Server 2008.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Top