Wanted: *Practical* Approach to Backups

[Scott Meyers]

I'm a backup believer, but I've had trouble finding a system I consider
practical. By "practical," I mean so easy to do, I'll do it all the time. This
rules out tapes, CDs, DVDs, etc., because replacing consumables is a hassle, and
hassles lead to not doing backups as often as I should. Besides, my collection
of old tapes, tape drives, and tape drive software (probably none of which will
work with my current system) demonstrates that, at least when it comes to
Windows-based tape systems for consumers, "archival" storage is really just a
couple of years.

I have a small LAN (2 machines, soon to be 3), and I think I know what I want.
I want a system where each week, a full backup of each machine is automatically
performed and written to a dedicated hard drive, probably an external one.
Here, "full backup" means everything on every hard drive, because Windows makes
it impossible to really separate data from programs (e.g., many programs write
into the registry or put config files in their program directories). Besides,
Windows programs are constantly being patched and updated, so the idea of
restoring software from its original media is fiction, anyway.

Each night, I want an incremental backup to be performed for each machine, again
with the backups being written to the dedicated hard drive for backups, and
again with the backup being applied to all files on each machine (i.e., an
incremental of all files on all drives, not just files rooted at some "data"
directory.

When the hard drive for backups gets full, I want the oldest backups to be
deleted to make room for new information, i.e., I want to use the dedicated
backup disk as a FIFO queue for backups. Assuming the size of the hard drive
for backups is a decent multiple of the total local disk usage on my machines
(easy in my case, as my 2 machines currently use only about 40GB of storage) and
assuming decent compression during backups, this should allow for several weeks
of backups for my system on a single hard drive.

I think the above is pretty straightforward, but I've had no luck finding
software that would (1) work across a LAN and (2) do a backup on open files
(e.g., mail files) while the system was running. (It's not practical for each
system to be shut down each night to do a backup. I have software running that
I don't want to shut down, e.g., my email client (which happens to be Eudora).

What I do now is manually make disk images about once a week. I'd like to do it
more often, but the software I run (Drive Image 2002 -- its successor, Drive
Image 7, won't install) requires a reboot to perform the imaging, and that's
very disruptive to the way I work.

Thanks for all help and suggestions.

Scott

 

[DL]

Depends how far your wallet stretches, you might want to look at NAS storage
solutions.
If you want a *full* incremental backup, perhaps Imaging software e.g.
Ghost, True Image etc, though they can also be used for data only.
I don't know if any backup software allows you to create incremental
backups, then when a disk is full, delete and start over, *automatically*.
You might also have problems with files 'in use' when you backup, as such
you may have to make some compromises.

 

[Scott Meyers]

Depends how far your wallet stretches, you might want to look at NAS storage
solutions.

I'm willing to spend a reasonable amount of money (up to, say, $500) to get
something that works, but my impression from the very cursory looks I've taken
at NAS options is that you're mostly buying hard drive space and networking.
The backup software still assumes that data is segregated from programs and that
files are never open. I'd love to be wrong about this characterization of NAS
products, so feel free to correct me.

As an example of how my data is not segregated from my programs, I'm running
Firefox, Thunderbird, FileZilla, and Eudora, each of which stores important data
(e.g., program configurations, extensions and *their* configurations, sent and
received mail and attachments, etc.) in the hierarchy where their respective
programs are stored. Furthermore, in my case, the only directory they have in
common is D:\ (I put only OS stuff on C:, and I don't use the "Program Files"
directory at all). Furthermore, at any time I'm typically working on a
half-dozen or so different projects, e.g., documents in Word or FrameMaker
format, Excel spreadsheets, Visual Studio projects, photos and other images,
etc. The files for these projects need to be backed up, but they're scattered
all over my hard drives, because I put them where they make the most sense to me
(e.g., associated with the directories housing the other files for the clients I
work with). This is contrary to the common "just backup your data folder, and
you'll be fine" advice I often see. I can't believe I'm the only person who
recognizes that important data is distributed in many different places and who
would like to have it backed up. What do other people do to address this problem?

If you want a *full* incremental backup, perhaps Imaging software e.g.
Ghost, True Image etc, though they can also be used for data only.

I have Drive Image 2002 (predecessor to Drive Image 7, which won't work on my
machine, and now they've both been "end of lifed" by Norton in favor of Ghost),
and it does only full images -- there's no support for incremental backups, as
far as I can tell. Do other imaging products support incremental images?

Thanks,

Scott

 

[Tim Judd]

I'm a backup believer, but I've had trouble finding a system I consider
practical. By "practical," I mean so easy to do, I'll do it all the
time. This rules out tapes, CDs, DVDs, etc., because replacing
consumables is a hassle, and hassles lead to not doing backups as often
as I should. Besides, my collection of old tapes, tape drives, and tape
drive software (probably none of which will work with my current system)
demonstrates that, at least when it comes to Windows-based tape systems
for consumers, "archival" storage is really just a couple of years.

I have a small LAN (2 machines, soon to be 3), and I think I know what I
want. I want a system where each week, a full backup of each machine is
automatically performed and written to a dedicated hard drive, probably
an external one. Here, "full backup" means everything on every hard
drive, because Windows makes it impossible to really separate data from
programs (e.g., many programs write into the registry or put config
files in their program directories). Besides, Windows programs are
constantly being patched and updated, so the idea of restoring software
from its original media is fiction, anyway.

Each night, I want an incremental backup to be performed for each
machine, again with the backups being written to the dedicated hard
drive for backups, and again with the backup being applied to all files
on each machine (i.e., an incremental of all files on all drives, not
just files rooted at some "data" directory.

When the hard drive for backups gets full, I want the oldest backups to
be deleted to make room for new information, i.e., I want to use the
dedicated backup disk as a FIFO queue for backups. Assuming the size
of the hard drive for backups is a decent multiple of the total local
disk usage on my machines (easy in my case, as my 2 machines currently
use only about 40GB of storage) and assuming decent compression during
backups, this should allow for several weeks of backups for my system on
a single hard drive.

I think the above is pretty straightforward, but I've had no luck
finding software that would (1) work across a LAN and (2) do a backup on
open files (e.g., mail files) while the system was running. (It's not
practical for each system to be shut down each night to do a backup. I
have software running that I don't want to shut down, e.g., my email
client (which happens to be Eudora).

What I do now is manually make disk images about once a week. I'd like
to do it more often, but the software I run (Drive Image 2002 -- its
successor, Drive Image 7, won't install) requires a reboot to perform
the imaging, and that's very disruptive to the way I work.

Thanks for all help and suggestions.

Scott

With windows based software? never seen it.

I doubt it'll be made for you, either -- you're being too specific (not
always a bad thing). Linux and BSD (Both derivatives of Unix), both
offer a backup solution, and I know there's one that pulls from the
client to backup, and stores on it's local drive.. I don't know the
name offhand, but we use it at work.

We've not needed to restore since I started (6 months ago), and
apparently, the Unix community supports it pretty well.

I favor BSD over Linux, but you may favor Linux over BSD when you try
them (suggestions: Ubuntu Linux [Ubuntu, Kubuntu, Xubuntu are all
flavors of Ubuntu], FreeBSD (installed at work, running beautifully).

We run Linux on our servers at work, so the product I'm mentioning may
work in Linux, but FreeBSD can run Linux binaries (via compatibility mode).

Good luck, keep us posted on your progress.

 

[DL]

I use True Image, and yes it does incremental backups/images, you can also
recover a selected file.
However I have my mail app closed when it backs up. I havent checked but I
think you'll find most backup software, of any type, will fail to backup a
file that is in use, allthough with some you can have it retry x times.

Assuming you would backup overnight, perhaps a script/task that closes apps
in use, backs up, then restarts the apps.
Personnaly I backup 'data' and certain settings off site every night, Image
every seven days and backup data/settings to an external drive every seven
days, but I'll probably invest in a NAS sollution at some stage.
I'm also using mirror raid + auto hot swap/rebuild

 

[David Webb]

FYI, Acronis True Image can backup system files and files in use. See "Expanded
Acronis Drive Snapshot technology" on this link:

http://www.acronis.com/homecomputing/products/trueimage/

It must work, because I've successfully restored both my Win2K Pro and WinXP Pro
systems many times, and they were backed up while Windows was running.

I'm running Acronis version 7.0, build 613.

| I use True Image, and yes it does incremental backups/images, you can also
| recover a selected file.
| However I have my mail app closed when it backs up. I havent checked but I
| think you'll find most backup software, of any type, will fail to backup a
| file that is in use, allthough with some you can have it retry x times.
|
| Assuming you would backup overnight, perhaps a script/task that closes apps
| in use, backs up, then restarts the apps.
| Personnaly I backup 'data' and certain settings off site every night, Image
| every seven days and backup data/settings to an external drive every seven
| days, but I'll probably invest in a NAS sollution at some stage.
| I'm also using mirror raid + auto hot swap/rebuild
|
|
| | > DL wrote:
| > > Depends how far your wallet stretches, you might want to look at NAS
| storage
| > > solutions.
| >
| > I'm willing to spend a reasonable amount of money (up to, say, $500) to
| get
| > something that works, but my impression from the very cursory looks I've
| taken
| > at NAS options is that you're mostly buying hard drive space and
| networking.
| > The backup software still assumes that data is segregated from programs
| and that
| > files are never open. I'd love to be wrong about this characterization of
| NAS
| > products, so feel free to correct me.
| >
| > As an example of how my data is not segregated from my programs, I'm
| running
| > Firefox, Thunderbird, FileZilla, and Eudora, each of which stores
| important data
| > (e.g., program configurations, extensions and *their* configurations, sent
| and
| > received mail and attachments, etc.) in the hierarchy where their
| respective
| > programs are stored. Furthermore, in my case, the only directory they
| have in
| > common is D:\ (I put only OS stuff on C:, and I don't use the "Program
| Files"
| > directory at all). Furthermore, at any time I'm typically working on a
| > half-dozen or so different projects, e.g., documents in Word or FrameMaker
| > format, Excel spreadsheets, Visual Studio projects, photos and other
| images,
| > etc. The files for these projects need to be backed up, but they're
| scattered
| > all over my hard drives, because I put them where they make the most sense
| to me
| > (e.g., associated with the directories housing the other files for the
| clients I
| > work with). This is contrary to the common "just backup your data folder,
| and
| > you'll be fine" advice I often see. I can't believe I'm the only person
| who
| > recognizes that important data is distributed in many different places and
| who
| > would like to have it backed up. What do other people do to address this
| problem?
| >
| > > If you want a *full* incremental backup, perhaps Imaging software e.g.
| > > Ghost, True Image etc, though they can also be used for data only.
| >
| > I have Drive Image 2002 (predecessor to Drive Image 7, which won't work on
| my
| > machine, and now they've both been "end of lifed" by Norton in favor of
| Ghost),
| > and it does only full images -- there's no support for incremental
| backups, as
| > far as I can tell. Do other imaging products support incremental images?
| >
| > Thanks,
| >
| > Scott
|
|

 

[Sid Knee]

As an example of how my data is not segregated from my programs, I'm
running Firefox, Thunderbird, FileZilla, and Eudora, each of which
stores important data (e.g., program configurations, extensions and
*their* configurations, sent and received mail and attachments, etc.) in
the hierarchy where their respective programs are stored.

Just as a fwiw, this is not necessarily true. Many programs do what you
say by default but with parameters added to the shortcut command line,
can be made to store their data, configuration info etc to a separate
location (and of course, retrieve from that location).

This is definitely true of Eudora since that's how I use it and have
done for years. I store it all in a separate data area which gets
regularly backed up to another machine on the network. Many other
(though regrettably not all) applications will allow this method of
working .... although you often have to dig for the command line
parameter info.

I don't know about Thunderbird and Firefox - my usage of those programs
is rather basic and the data integrity is relatively unimportant to me
so I haven't bothered. I did at one time do this extensively with
Netscape though and I think it's probable that Thunderbird and Firefox
will allow the same method. I do recall, in the case of Netscape, that
although the parameters were simple, it took considerable digging to
find out how to do it. The end result is worth it though.

 

[Scott Meyers]

Just as a fwiw, this is not necessarily true. Many programs do what you
say by default but with parameters added to the shortcut command line,
can be made to store their data, configuration info etc to a separate
location (and of course, retrieve from that location).

I am aware of this, but there are three problems. First, not all programs offer
this flexibility. As far as I know, for example, there is no way to tell
FrameMaker to look for maker.ini (its configuration file) anyplace other than
the program directory. Second, even in cases where it's possible, I don't
currently have things organized that way, and it may not be practical to change
now. For example, Eudora hard-codes the path to attachments in mailbox files,
so moving the attachments directory to some other location would break the links
between messages and attachments. Third, it just strikes me as ridiculous to
organize my data in a way that makes more sense to my backup program than it
does to me. The machine is supposed to be working for me, not vice versa.

Having said all that, is this really what people do in practice: jump through
hoops of file to coerce all their programs to put their configuration and user
data in a special "data" directory so that it can be easily backed up?

Thanks,

Scott

 

[Scott]

Have a look at EMC Retrospect 7.5.
I think it does allot of what you want.

 

[Sid Knee]

I think you are re-iterating many of my points, though from a "glass
half empty" perspective:

....... As far as I know, for example, there is no way to tell FrameMaker > to look for maker.ini (its configuration file) anyplace other than the program
directory.

In my experience "as far as I know" is the operative phrase in many
cases. Like I say you sometimes have to dig. If you want to find them of
course.

For example, Eudora hard-codes the path to attachments in mailbox files,
so moving the attachments directory to some other location would break the links
between messages and attachments.

It simply doesn't work that way for me (Eudora versions 5, 6 or 7). My
attachments directory is located in the (non-default) data area along
with the the mailbox files and there is no mail/attachment link problem.

it just strikes me as ridiculous to organize my data in a way that
makes more sense to my backup program than it does to me. The machine
is supposed to be working for me, not vice versa.

Well, your original assertion was that the apps wouldn't *allow* data to
be organised in this fashion. I was merely pointing out that that isn't
necessarily true. Fact is though that this method of data organisation
is a fairly common one because it makes sense to many *people* not just
a backup program.

Having said all that, is this really what people do in practice: jump
through hoops of file to coerce all their programs to put their
configuration and user data in a special "data" directory so that it can
be easily backed up?

I agree with your sentiments that computers and programs should work for
people. But, for myself, I don't consider that this method of organising
data is "jumping through hoops". To me (and I believe to many others)
it's one logical method.

In the end though, it seems that what you are looking for is not simply
a program that automates backups perhaps more than the current offerings
do - but one that is also closely tailored to your "style". Albeit, that
style may differ widely from the mainstream. There is a solution ....
it's called custom software.

 

[Scott Meyers]

Have a look at EMC Retrospect 7.5.
I think it does allot of what you want.

Thanks very much for the pointer. From what I can tell from the literature, it
does everything I want (including automatic deletion of old backups to make room
for new ones) except backing up open files, and if I were running WinXP instead
of Win2K, it'd do that, too. If it works as advertised, it will be a very
attractive product. List price is $120, and there's a 30-day free trial, which
I'm downloading now.

Does anybody have any experience with EMC Retrospect 7.5 for Windows
Professional Edition? (There's also an Express Edition, which is apparently
widely OEMed and, as to be expected, significantly reduced in functionality.)

Scott