Huge data file

[CSharper]

I have a data file which is 2.1GB (zipped), when the exe runs, it
copies the content of the zip file to a destination folder. What is
the best way to achive this? Current implementation is to zip the file
at the source and at the destination unzip the file and if the file
dates are different then copy the files over. Is there a better way to
do this other than zipping and unzipping?
Thanks.

 

[Arne Vajhøj]

I have a data file which is 2.1GB (zipped), when the exe runs, it
copies the content of the zip file to a destination folder. What is
the best way to achive this? Current implementation is to zip the file
at the source and at the destination unzip the file and if the file
dates are different then copy the files over. Is there a better way to
do this other than zipping and unzipping?

You can not avoid zipping and unzipping if you use a zip file.

You can avoid any intermediate files, if your app checks
dates and if necessary extract the file directly to final
destination.

Arne

 

[Marc Gravell]

Can I ask what is in the data file? Not specifics - but is it
* an archive of lots of other files
* raw binary data
* serialized .NET entities, such as xml

In the first bullet, you could perhaps read the archive with tools like
#ZipLib and only extract / copy the interesting files - but note that
you are on the limit of what this will work with. Alternatively, tools
like "robocopy" will work on the *extracted* files (not the single
archive) to efficiently copy on the changed data; you can use
OS/file-system compression rather a zip to allow this to work happily.

In the second bullet; well - that is tricky; the data is likely to be
fairly dense, so I'd be surprised if it compressed very well anyway...

In the third bullet - perhaps try an alternative data format to reduce
space? I have lots of current ideas in this area if you are interested...

Marc

 

[Ken Foskey]

I have a data file which is 2.1GB (zipped), when the exe runs, it copies
the content of the zip file to a destination folder. What is the best
way to achive this? Current implementation is to zip the file at the
source and at the destination unzip the file and if the file dates are
different then copy the files over. Is there a better way to do this
other than zipping and unzipping? Thanks.

You could look at the application rsync to allow synchronisation of files
that only change a little bit each time.

If you are talking a Database (eg Access) then look at replication.

Ta
Ken

 

[CSharper]

You could look at the application rsync to allow synchronisation of files
that only change a little bit each time.

If you are talking a Database (eg Access) then look at replication.

Ta
Ken

Thanks to you all. I really appriciate your comments.