Is Dump Complete?

[Guest]

We have a file that is an extract file that represents data from several
different systems (main frame, etc). This data varies and so from day to day
the file that gets generated has variant size of data. Sometimes is just 5
meg, other times is can be 10 gig. This means that the time it takes for the
file to be completed varies also. In other words it is un-predictable what
the size will be.

Now, the problem is, when the file is complete we need to kick off another
process that parses the file as well as other tasks that are not related but
can not kick off until the file is done being generated.

So, how do we know the file is done being written? The app that generates
the file is a 3rd party and is no-longer supported by the vendor.

I've considered writting code that will attempt to open the file and if the
app still has a lock on it then an exception will get thrown and voila we
know it isn't done yet. That however, is un-acceptable jerry rigging. This
needs to be production worthy code and done properly.

So my question to you guys, in C#, what sort of solution would you provide
the problem of knowing when the file is done?

 

[Willy Denoyette [MVP]]

| We have a file that is an extract file that represents data from several
| different systems (main frame, etc). This data varies and so from day to
day
| the file that gets generated has variant size of data. Sometimes is just 5
| meg, other times is can be 10 gig. This means that the time it takes for
the
| file to be completed varies also. In other words it is un-predictable what
| the size will be.
|
| Now, the problem is, when the file is complete we need to kick off another
| process that parses the file as well as other tasks that are not related
but
| can not kick off until the file is done being generated.
|
| So, how do we know the file is done being written? The app that generates
| the file is a 3rd party and is no-longer supported by the vendor.
|
| I've considered writting code that will attempt to open the file and if
the
| app still has a lock on it then an exception will get thrown and voila we
| know it isn't done yet. That however, is un-acceptable jerry rigging. This
| needs to be production worthy code and done properly.
|

Unfortunately it's the only way to handle this, you try to open the file, if
it fails because of a sharing violation, you wait a couple of seconds and
retry, what makes you think that this isn't production worthy?
To handle this more "nicely", you need two cooparating applications, one
producer and one consumer. The producer uses a semaphore or mutex to signal
the consumer that he's has finished producing the data, the consumer wait's
for the semaphore to be signaled.
The real professional solution is to have both consumer and producer to run
simultaniously, this by using region locks or semaphores to signal data
ready to be consumed.

Willy.

 

[Jack Robertson]

In addition to what Willy said (which is absolutely correct), is the file so
tightly controlled that you can guarantee no 3rd-party will ever access it
besides the programs that are supposed to (e.g., can a user unexpectly open
it up, can it be accessed via some backup program that runs when not
expected, etc.). It's unlikely so a sharing violation can potentially happen
when you're not expecting it. Morevoer, since you can't synchronize access
given that it's created by a 3rd-party vendor (using the methods recommended
by Willy), what other choice do you have but to check for sharing
violations? To do it more efficiently however (rather than explicitly
checking every few seconds), perhaps you can leverage a class like
"FileSystemWatcher" (though I've never had a need to use it myself). Watch
the file in question IOW. If no activity has taken place for a given period
of time then you may be able to assume the file is now free (so try to a
open it exclusively presumably and if it fails because of a sharing
violation then start your wait over again). Food for thought anyway.