How do I check for complete file

[Anders Eriksson]

Hello!

I'm trying to create a program that will watch a directory and when a file
is created print that file. I have used FileSystemWatcher for watching the
directory and I get an created event.

The problem is that I don't know when the file is complete when using
Explorer copy or move.

I have tried to use FileIOPermission but I always get that the file is
unlocked.

How do I check for file complete?

// Anders
--
English isn't my first, or second, language.
So anything rude or strange are due to the translation

private static bool isFileLocked(string path)
{
FileIOPermission _available = new FileIOPermission( PermissionState.None);
// blank slate
try
{
_available.SetPathList( FileIOPermissionAccess.Write, path);
_available.Demand( ); // does file permit shared writing?
Console.WriteLine("isFileLocked: false ");
return false;
}
catch ( SecurityException _se)
{
Console.WriteLine("isFileLocked: true ");
return true;
}
}

 

[Scott]

I have never, personally needed to do this before, though I remember seeing
other messages about this in one of these newsgroups a while back.

As I remember it, one of two things can be done:
1- actually try to open the file for writing (though i don't know how this
is different than what you did)
2- periodically check the file size to see if it is still increasing (ie. in
the process of being written)

sorry i can't be more help than that...

scott

 

[Peter Bromberg [MVP]]

The cleanest way I have found to handle this (assuming that you are able to
do it in your situation) is to transmit a small control file to the folder
containing the name and or filetime of the real file to be transferred.
When the real file transfer is complete, you remove the small control file.
So all you would need to do is check for the presense of the control file.
Peter

 

[Guest]

I know that this doesn’t fix your problem however could you copy/move the
file to another folder on the same drive as the directory you are monitoring,
and once that operation is complete have the copier/mover make one final move
to your monitoring directory?

The nice thing about moving files on the same disk is that it is just a
pointer update as far as the system is concerned, the file does not change
locations physically so there is no delay between its first appearing in the
directory and it’s being fully accessible.

Brendan

 

[Guest]

It might be very hairy, and there are certainly some drawbacks but...

Take a look at FSCTL_READ_USN_JOURNAL and READ_USN_JOURNAL_DATA which can be
used with DeviceIoControl() to get NTFS change journal records. This
structure has a flag for specifying only returning with the change journal
records once the file is closed.

So in theory...

1. std notification from FileSystemWatcher.
2. acquire necessary info to build a READ_USN_JOURNAL_DATA structure
3. call into DeviceIoControl() with ReturnOnlyOnClose set
4. discard results... at this point copy is done

This should work for copy, don't know about move.

Better yet, I think FileSystemWatcher is implemented using some of the NTFS
change journal magic... so look at extending it. Maybe by spelunking the IL
for FileSystemWatcher will help with understanding how MS did it.

 

[Willy Denoyette [MVP]]

It might be very hairy, and there are certainly some drawbacks but...

Take a look at FSCTL_READ_USN_JOURNAL and READ_USN_JOURNAL_DATA which can
be
used with DeviceIoControl() to get NTFS change journal records. This
structure has a flag for specifying only returning with the change journal
records once the file is closed.

So in theory...

1. std notification from FileSystemWatcher.
2. acquire necessary info to build a READ_USN_JOURNAL_DATA structure
3. call into DeviceIoControl() with ReturnOnlyOnClose set
4. discard results... at this point copy is done

This should work for copy, don't know about move.

Better yet, I think FileSystemWatcher is implemented using some of the
NTFS
change journal magic... so look at extending it. Maybe by spelunking the
IL
for FileSystemWatcher will help with understanding how MS did it.

No, the filesystem watcher does not rely on NTFS and NTFS Journaling. It
uses ReadDirectoryChangesW,FindFirstChangeNotification and
FindNextChangeNotification as it must be usable on downlevel platforms too.

Willy.

 

[Guest]

No, the filesystem watcher does not rely on NTFS and NTFS Journaling. It
uses ReadDirectoryChangesW,FindFirstChangeNotification and
FindNextChangeNotification as it must be usable on downlevel platforms too.

Willy.

Good call.

Even so, would hooking the NTFS change journal be a good solution for this
type of problem. It seems as though a person is having to dig pretty deep
just to find out when a file got closed...

If it might be a reasonable idea, would it even work?

 

[Bruce Wood]

Yes... I was going to suggest a rename as the simplest solution, if you
have control over the file's producer.

My (traditional) solution has been this:

1. Producer writes file to target directory under a temporary name.
2. After flushing and closing the file, producer renames file to new
name, indicating that file is complete.
3. Consumer detects new file with desired name and immediately renames
file to a different temporary name (this is necessary only if you scale
up to multiple consumers). If rename is successful, consumer processes
file under temporary name.

Of course, this works only if you have control over the producer. If
you don't, then you have to resort to one of the fancier solutions
proposed here.

 

[Willy Denoyette [MVP]]

Good call.

Even so, would hooking the NTFS change journal be a good solution for this
type of problem. It seems as though a person is having to dig pretty deep
just to find out when a file got closed...

If it might be a reasonable idea, would it even work?

Well, I have done something lately with the NTFS change journal, not exactly
for the same purpose though, I had it working but I dropped it in favor of a
FS Filter Driver. The reason I didn't use change journal was for reasons of
maintenance issues with the journal streams. While it might be a workable
solution, there is an other options called ETW (Event Tracing for Windows)
which is event driven and provides very low overhead (by the way this is the
thing used by the VS2005 trace facility).

Willy.

 

[Anders Eriksson]

As I remember it, one of two things can be done:
1- actually try to open the file for writing (though i don't know how this
is different than what you did)

I have now changed so I use the Win32 API CreateFile() and now got it to
work! To make this work I need to compile the program with /unsafe switch.

Is there a C# function that does the same as CreateFile?

2- periodically check the file size to see if it is still increasing (ie. in
the process of being written)

It seems like the size of the file is written when the file is created,
before the contents is copied.

// Anders

 

[Per Salmi]

Hello Daymon,

Are you sure that the journal records approach will work?

I have a scenario where we watch a network share and the "check for complete
file"
behaves differently when the file is written to the share by Windows clients
and Mac OS X
clients. Macs seem to close the file and reopen it multiple times so it results
in false alarms
for a completed file... So I hope the journaling thing will work.

/Per Salmi

 

[Per Salmi]

Hello Anders,

On Tue, 4 Oct 2005 16:41:56 +0200, Scott wrote:
I have now changed so I use the Win32 API CreateFile() and now got it
to work! To make this work I need to compile the program with /unsafe
switch.

Is there a C# function that does the same as CreateFile?

In what way is CreateFile better to find out if the file is complete?

It seems like the size of the file is written when the file is
created, before the contents is copied.

That is true when a Windows Explorer is used to copy a file. I have seen
a different behaviour
when a Mac client copies a file to a folder. Then the file size increases
with each change notification
event that occurs when I have a FileSystemWatcher setup on the folder.

/Per Salmi

 

[Anders Eriksson]

Hello Per,

In what way is CreateFile better to find out if the file is complete?

In the way that CreateFile will return an error when the copying is going
on. FileIOPermission will not!

I'm just guessing here but I think the difference is that CreateFile will
check for 'Share Exclusive' and when Explorer is copying then CreateFile
can't open the file Exclusive and gives an error!

That is true when a Windows Explorer is used to copy a file. I have seen
a different behaviour when a Mac client copies a file to a folder. Then
the file size increases with each change notification event that occurs
when I have a FileSystemWatcher setup on the folder.

(.Fortunately.) I only have to worry about Windows ;-)

// Anders