Robocopy

[Simon]

We make extensive use of Microsoft's Robocopy to secure
files and folders and have done for several years. We
have recently introduced a new Progress based application
and have discovered conflict between the two. By running
Robocopy against the 'live' application area we have
discovered corruption to source files - I know this sounds
implausible but these are source file that are corrupted.
The assumption is that Robocopy takes out 'write locks' on
source files that the application is unable to contend
with. We obviously accept that a destination file may at
times be inconsistent.

Robocopy version 1.96 is being used, provided with the
Windows 2000 resource kit, (supplement 1). The Robocopy
switches in use are '/mir /r:3 /w:3 (that's mirror, 3
retries, wait 3 seconds). We are currently evaluating
Pixelab's XXCOPY Pro to perform the same function with
more promising results. Strategically this is an
important issue to us. Any copy program capable of making
the source file inconsistent is worrying, especially since
it's used extensively. Have any members experienced
anything similar. Any comments will be gratefully
received.

 

[John LeMay]

By running Robocopy against the 'live' application area we have discovered
corruption to source files -

Not that it helps you much, but I've seen the same thing happen with
another application in a similar scenario. Trying to "image" the live data
causes corruption. I found no way to fix the issue other than to make sure
the application is not running when we try to run the process. This is one
of the reasons I really prefer Unix bases systems sometimes!

--
John LeMay
kc2kth
Senior Technical Manager
NJMC | http://www.njmc.com | Phone 732-557-4848
Specializing in Microsoft and Unix based solutions

 

[Kan Yabumoto]

We make extensive use of Microsoft's Robocopy to secure
files and folders and have done for several years. We
have recently introduced a new Progress based application
and have discovered conflict between the two. By running
Robocopy against the 'live' application area we have
discovered corruption to source files - I know this sounds
implausible but these are source file that are corrupted.
The assumption is that Robocopy takes out 'write locks' on
source files that the application is unable to contend
with. We obviously accept that a destination file may at
times be inconsistent.

Simon:

I'm not sure if Robocopy can ignore a lock made by a
running program which still keeps the file open
(I doubt it but then, I don't know what Robocopy really
does.)

In the case of XXCOPY, it opens the source file with both
FILE_SHARE_READ and FILE_SHARE-WRITE enabled (in default mode).

In most cases, when an application really wants to insist
on sharing the file (which was intended to be written/
modified) with other programs (such as XXCOPY),
the application should lock the file (and keep it locked
as long as the entire file is not coherent) while the file
contents are being modified, in my opinion.

XXCOPY does not try to manipulate the share-mode setting
or file lock situation on a file which has been opened
by another application. It simply tries to access (of
course, for read-only purposes) the file and if the lock
is on or file-share is not permitted, it simply waits
and retries (for the pre-determined retry-period).

As you have noted, accessing a file which is currently
being opened (for write) (and apparently at least the
FILE_SHARE_READ setting is provided by the original
application), it takes chances of accessing the file
which may not be wholly coherent. At the least,
at the end of the file-copy operation, XXCOPY checks
the file size of the newly copied file against the
original (by the default /V switch). You may further
increase the dependability by adding the /V2 switch
which ensures that the file in the destination is
identical to the source file on a byte-by-byte
comparison. If any byte is not consistent, the new file
will be discarded (and the file copy is treated as
a failure).

But, in a worst case scenario, even if the byte-by-byte
check passes its test, there is no guarantee that the
original application was not in the middle of modifying
a part of the file and more changes are pending at the
time to make the file to be eventually coherent from
the first byte to the last.

If you are concerned with the possibility of this
worst-case scenario and furthermore, if such a case is
unacceptable to you, I suggest that you use the /SH0
switch in XXCOPY which makes sure that file access is in
exclusive (non-sharing) fashion only. This will inevitably
make many more failures in file copy.

I suppose in nearly all such cases, the user is generally
aware of a particular (e.g., database) application which
keeps a handful of huge files open at nearly all the time.
Assuming this is the case, I suggest that you use the
/SH0 switch on a separate, more targeted file copy
operation. In other words, I do not recommend that you
copy the entire volume for backup using the /SH0 switch.
Rather, you should exclude certain problematic files
(e.g., the database files) from the general backup
operation and execute a separate XXCOPY command just for
the shared files which are known to have problems in
sharing.

In conclusion, I'm not sure if there is a bullet-proof
algorithm in accessing a file which is being modified by
an application and expect the resultant file to be
always right (it requires that the application which
writes the file needs to behave correctly).

But, with XXCOPY, /V2 and /SH0 are worth experimenting.

Kan Yabumoto
The author of Xxcopy.