2Tb Offsite Backup?

[rpeters+groups]

We are looking for a solution that allows us to accomplish the
following:
- Weekly full backups (approx. 50 hours available)
- Daily incremental backups (approx 10 hours available)
- On-site access to current week's backup
- Offsite storage of previous 2 weeks

We have the following resources we would appreciate being able to re-
use:
- 4x 750Gb SATA drives
- 1x Hot-Swap SATA bay
- Windows Backup software

Our current solution is as follows, and works great for up to approx
700Gb of data:
- Windows Backup is scheduled as described
- We swap the single attached 750Gb drive weekly, taking it off site
- We have a 3-disk rotation, guaranteeing that an on-site disaster
only loses at most one week of data. (One disk is always off-site, a
second is brought in on swap-day, and the third is live.)
- One backup disk sits in reserve in case of failure.

Unfortunately, our storage demands have increased past the amount that
can be stored on a 750Gb, and we need to evaluate and find a new
solution.

What products / methods should we be looking at?

Thanks for any help.
ICOM Productions, Inc.

 

[Rod Speed]

We are looking for a solution that allows us to accomplish the
following:
- Weekly full backups (approx. 50 hours available)
- Daily incremental backups (approx 10 hours available)
- On-site access to current week's backup
- Offsite storage of previous 2 weeks

We have the following resources we would appreciate being able to re-
use:
- 4x 750Gb SATA drives
- 1x Hot-Swap SATA bay
- Windows Backup software

Our current solution is as follows, and works great for up to approx
700Gb of data:
- Windows Backup is scheduled as described
- We swap the single attached 750Gb drive weekly, taking it off site
- We have a 3-disk rotation, guaranteeing that an on-site disaster
only loses at most one week of data. (One disk is always off-site, a
second is brought in on swap-day, and the third is live.)
- One backup disk sits in reserve in case of failure.

Unfortunately, our storage demands have increased past the amount that
can be stored on a 750Gb, and we need to evaluate and find a new solution.

What products / methods should we be looking at?

One approach would just be to change over to using the 1TB drives now available.

 

[Smite]

We went from 400Gb of data to 750Gb within a matter of months, and are
looking for a longer-term solution, so simply gaining 250Gb of storage is
not enough. I should have mentioned it in the body of the message as well,
but we want something capable of at least 2Tb. (Though, even better if
it's also capable of using a pair of 750Gbs to acheive 1.5Tb storage until
we decide to purchase larger / more drives.)

Thanks.

 

[Rod Speed]

Rod Speed <[email protected]> wrote

We went from 400Gb of data to 750Gb within a matter of months,

What is the nature of those files, video presumably ?

If so, a more intelligent approach to backup would likely be more
viable than the simplistic approach you are currently taking.

and are looking for a longer-term solution, so simply gaining 250Gb of storage is not enough. I should have mentioned
it in the body of the message as well, but we want something capable of at least 2Tb.

One obvious approach is a pair of 1TB drives in a spanned
config as the destination for the weekly backup which is
presumably the only one you actually need the 2TB capacity for.

(Though, even better if it's also capable of using a pair of 750Gbs to acheive 1.5Tb storage until we decide to
purchase larger / more drives.)

That spanning approach would allow that very conveniently.

You'd just need to add another hot swap bay hardware wise.

I'd personally have the pair of drives in an external esata enclosure instead.

 

[Smite]

The files are largely a mix of video, photo, and flash files, yes.
Any suggestions on "a more intelligent approach" would be very much
appreciated. Cost is likely to be a driving factor however, explaining the
current setup.

Agreed, an external eSATA enclosure capable of hotswapping and spanning
disks seems like a good solution, Do you have any suggestions on one that
would meet our needs? And one that exceeds them, so that we are
future-proofing the investment?

 

[Rod Speed]

The files are largely a mix of video, photo, and flash files, yes.
Any suggestions on "a more intelligent approach" would be very much appreciated.

Just write the new file to more than one backup destination, with at least one off site.

No need to keep continually backing those up weekly.

Cost is likely to be a driving factor however, explaining the current setup.

It wouldnt cost any more to do the backup more intelligently.

Agreed, an external eSATA enclosure capable of hotswapping and spanning disks seems like a good solution, Do you have
any suggestions on one that would meet our needs?

Any external esata multi drive housing should do
it fine with the spanning done at the OS level.

And one that exceeds them, so that we are future-proofing the investment?

Its just as easy to have say 3 drives in the housing instead of 2 etc.

 

[Smite]

Our only "offsite" access is via our Internet connection, which is capped
at a prohibitive level for using it for backup purposes, so we'll need to
stick to physical media swapping for that purpose.

The data is important enough to warrant repetition, so the weekly full
backup still seems to me like a must as well.

To be clear, we are talking about a very large number of files altered and
created very frequently by 30-40 people, so the entire process must be
entirely transparent to them. Hence a need for a scheduled sync/backup and
not just saving to multiple locations.

Also to be clear, despite cost being a major concern, getting it right is
more important. So I'm not above suggestions to move to more complex
systems either, such as a XRAID box with mirrored RAID 5 setups, if
necessary. (But without any experience with them, I'm not sure that would
be any better than the external eSATA enclosure mentioned. Hence the
request for opinions. )

Thanks for your help,
ICOM Productions.

 

[Arno Wagner]

We are looking for a solution that allows us to accomplish the
following:
- Weekly full backups (approx. 50 hours available)
- Daily incremental backups (approx 10 hours available)
- On-site access to current week's backup
- Offsite storage of previous 2 weeks

We have the following resources we would appreciate being able to re-
use:
- 4x 750Gb SATA drives
- 1x Hot-Swap SATA bay
- Windows Backup software

Our current solution is as follows, and works great for up to approx
700Gb of data:
- Windows Backup is scheduled as described
- We swap the single attached 750Gb drive weekly, taking it off site
- We have a 3-disk rotation, guaranteeing that an on-site disaster
only loses at most one week of data. (One disk is always off-site, a
second is brought in on swap-day, and the third is live.)
- One backup disk sits in reserve in case of failure.

Unfortunately, our storage demands have increased past the amount that
can be stored on a 750Gb, and we need to evaluate and find a new
solution.

What products / methods should we be looking at?

Your backup procedure looks reasonable to me. As you are happy with
it, you should increase backup space. One possible approach would be
to use a pair of the drives in RAID0 or JBOD mode. Best do this with a
hardware RAID controller, that supports swapping out the whole
array. No idea which ones do, but you may want to ask Arcea and 3ware
tech support about this. Otherwise you would have to cold-swap the
drive-pairs, i.e. shut down, swap, reboot. Hot-swapping a whole RAID1
may also be possible with Windows software RAID (I think they are
called ''dynamic disks''). I know that it is possible under Linux with
SATA controllers supporting hot-swap.

Arno

 

[Rod Speed]

Our only "offsite" access is via our Internet connection, which is
capped at a prohibitive level for using it for backup purposes,

Is that really so if you just write a new file offsite the once ?

so we'll need to stick to physical media swapping for that purpose.

OK, but you can still do the backup more intelligently
than just moving the entire collection offsite every week.

The data is important enough to warrant repetition, so the weekly full backup still seems to me like a must as well.

To be clear, we are talking about a very large number of files
altered and created very frequently by 30-40 people, so the entire
process must be entirely transparent to them. Hence a need for a
scheduled sync/backup and not just saving to multiple locations.

Sure, but its still possible to do it more intelligently than
just a crude total copy of everything weekly, so it may
well be possible to do the offsite via the net connection.

Also to be clear, despite cost being a major concern, getting it right is more important. So I'm not above suggestions
to move to more complex systems either, such as a XRAID box with mirrored RAID 5 setups, if necessary.

The problem isnt the daily backups, you'd
still need the weekly full backup with RAID.

(But without any experience with them, I'm not sure that would be any better than the external eSATA enclosure
mentioned.

I dont believe it would be myself.

Hence the request for opinions. )

Too radical |-(

Thanks for your help,
ICOM Productions.

No problem, thats what these technical newsgroups are for.

 

[rpeters+groups]

Our daily incrementals are generally around the 1-2Gb mark, our
monthly cap is 150Gb, and we're already using most of that. So yes,
transferring those off-site via Internet each day is cost prohibitive.
(As in, it would at least quadruple our monthly bill.)

So, leaving the 'net out of things, what are some "more intelligent"
options?

 

[Rod Speed]

Our daily incrementals are generally around the 1-2Gb mark,

That one is irrelevant because the current hardware handles that fine.

our monthly cap is 150Gb, and we're already using most of that. So
yes, transferring those off-site via Internet each day is cost prohibitive.

I wasnt suggesting doing that with the daily incrementals,
because the current hardware handles them fine.

(As in, it would at least quadruple our monthly bill.)

Not if you do it intelligently so that only stuff that
isnt changing anymore is moved offsite that way.

So, leaving the 'net out of things,

I still dont think you can do that.

what are some "more intelligent" options?

Basically instead of mindlessly moving the entire collection
weekly, move just the stuff that hasnt changed in a say a
week offsite to a permanent offsite repository and not
repeately moving it offsite weekly once its stopped changing.

That should drastically reduce the volume of the weekly backups and
it may well then be feasible to continue with the current hardware.

 

[Smite]

Ok, this is going to need more explaining before I understand.

Is your suggestion then to have an offsite site with an existing baseline,
and only send over diffs once a week?

The reason for the "mindless" move of the entire collection is to ensure
that we have redundancy, and that should any single drive fail, we can
still recreate the entire file base, even if somewhat out of date. I
assume your method maintains this redundancy through RAID on the offsite
end?

Currently, the offsite storage has no functionality, no way to validate
the baseline data, and drives do go corrupt over time. How does your
solution ensure everything is successfully backed up at all times, without
requiring the same hardware we would need to extend our current setup?

Obviously, the other option is to have 3 identical "base" drives stored
somewhere, and just build constant incrementals based off of them, but
eventually those incrementals will add up to more data than a new full
backup, rendering that solution pointless.

If our incrementals are still done the old way, and our base is stored
off-site at all times... Then what's the 'net being used for?

If I understand correctly, I'm afraid I don't see the benefit, so I'll
assume you meant something else. Unfortunately, I wasn't able to decipher
it based on your comments so far. Would you mind spelling out exactly
what's involved on each end, and what's used to transfer what data at each
point?

Thank you.

 

[Rod Speed]

Ok, this is going to need more explaining before I understand.

Is your suggestion then to have an offsite site with an existing baseline, and only send over diffs once a week?

No. Instead of mindlessly moving the entire collection offsite every
single week, and doing that repeatedly for the entire collection,
every week, once a particular file isnt being changed anymore,
stop moving it offsite weekly once its already got offsite somehow.

That way it may well be feasible to move a particular file that isnt
being changed anymore, offsite using the net, because you dont
keep moving it offsite weekly once its already in the offsite collection.

The reason for the "mindless" move of the entire collection is to
ensure that we have redundancy, and that should any single drive
fail, we can still recreate the entire file base, even if somewhat out of date.

Sure, but once its offsite somehow, there is no need to continually
mindlessly move the stuff thats ALREADY got offsite, offsite
weekly. The intelligence comes from keeping track of what
has got offsite, and so doesnt need to be copied offsite again.

That way the volume being moved offsite may well become low
enough to be viable via the net and so in effect you only move stuff
offsite when its got changed, not mindlessly every week forever.

I assume your method maintains this redundancy through RAID on the offsite end?

You can do whatever you want in that regard, its a separate issue
to how you get the volume moving offsite down so that doing that
via the net becomes economically viable since it only ever moves
over that link once, not repeatdly every week forever.

You can make a case that you dont actually need RAID on
the offsite end, since its a duplicate of what you have onsite.

Currently, the offsite storage has no functionality, no way to validate the baseline data, and drives do go corrupt
over time. How does your solution ensure everything is successfully backed up at all times, without requiring the same
hardware we would need to extend our current setup?

Its quite different hardware and doesnt have the limitations on
what is available in an external housing when you do it intelligently.

And you eliminate the risk of what is in the external housing getting
damaged in transit to the offsite site when the files are moved using the net.

Obviously, the other option is to have 3 identical "base" drives
stored somewhere, and just build constant incrementals based off of them, but eventually those incrementals will add
up to more data than a new full backup, rendering that solution pointless.

If our incrementals are still done the old way, and our base is stored off-site at all times... Then what's the 'net
being used for?

To get the files offsite in the first place, and that is
done once, instead of mindlessly weekly forever which
is a problem with using the net for that cost wise.

If I understand correctly, I'm afraid I don't see the benefit, so I'll assume you meant something else. Unfortunately,
I wasn't able to decipher it based on your comments so far. Would you mind spelling out exactly what's involved on
each end, and what's used to transfer what data at each point?

The system is the same as you currently have, for the daily backups.

The difference is that stuff is only moved offsite once, when it changes,
and itsnt moved offsite again unless a particular file changes in the future.

All that really needs is some way of keeping track of what has got offsite
so you can move it offsite again just once if it changes in the future.

 

[Smite]

While I appreciate the time you have taken on this, I'm forced to conclude
you don't understand what it is you're suggesting.
I CAN NOT move data via the 'net. The volume of new content alone is too
high.
There is no magical system "offsite" that stores data without hardware,
and buying the hardware certainly is more expensive than a 4-bay NAS.
You're suggesting that I NOT backup (offsite) the most vulnerable data of
all: The data that's currently being altered / created. This is just plain
stupid. Waiting until something is finalized is not an acceptable backup
strategy, and we have a separate archival system in place for that.
Again, redundancy is NOT "mindless", and is a vital part of any backup
strategy.

If you have any other suggestions that keep these facts in mind, I would
still appreciate hearing them.

 

[Rod Speed]

While I appreciate the time you have taken on this, I'm forced to conclude you don't understand what it is you're
suggesting.

Corse I can understand it. Its you that cant understand the basics yourself.

I CAN NOT move data via the 'net. The volume of new content alone is too high.

Thats hard to believe, whatever you claim.

And its relevant to whether a more intelligent approach to the WEEKLY
backups, not just mindlessly moving the entire collection to the offsite
repeatedly every single week, repeatedly forever, solves the problem
with the fact that your current hardware should be fine to move JUST
the changed stuff weekly once only.

There is no magical system "offsite" that stores data without hardware,

Duh. No one ever said there was.

and buying the hardware certainly is more expensive than a 4-bay NAS.

You need more than JUST a single 4 bay
NAS to provide an adequate offsite backup.

You're suggesting that I NOT backup (offsite) the most vulnerable data of all: The data that's currently being altered
/ created.

Wrong. YOU CONTINUE TO USE WHAT YOU ARE CURRENTLY USING
TO DO THAT. THE PROBLEM IS THAT THE CURRENT HARDWARE
CANT COPE WITH THE HIGHER VOLUME THAT THE MINDLESS
REPEATED WEEKLY BACKUPS INVOLVE, BECAUSE YOU ARE
MINDLESSLY COPYING THE SAME STUFF TIME AFTER TIME AFTER
TIME EVERY SINGLE WEEK, WHETHER ITS CHANGED OR NOT.

IF YOU HAVE A SINGLE SYSTEM OFFSITE THAT CAN HOLD THE
ENTIRE COLLECTION, AND JUST USE THE 750GB DRIVES TO MOVE
JUST THE STUFF THATS CHANGED IN THE WEEK TO THE OFFSITE
SYSTEM, THEN IT SHOULD BE BIG ENOUGH TO ALLOW THAT.

This is just plain stupid.

Having fun thrashing that straw man are you ?

Waiting until something is finalized is not an acceptable backup strategy,

No one ever said it was.

and we have a separate archival system in place for that.

So if you continue to use that for that, that doesnt affect THE
CHANGED APPROACH TO THE WEEKLY OFFSITE BACKUPS.

Again, redundancy is NOT "mindless",

No one ever said it was. WHAT IS MINDLESS IS REPEATEDLY MOVING
THE TOTAL COLLECTION EVERY SINGLE WEEK TO THE OFFSITE
AND THAT IS WHERE YOU HAVE A VOLUME PROBLEM.

and is a vital part of any backup strategy.
Duh.

If you have any other suggestions that keep these facts in mind, I would still appreciate hearing them.

I'm not going to bother, you are clearly so stupid that you
cant even manage to comprehend exactly what is being
proposed and I'm not going to waste my time trying to
get it thru your thick skull exactly what is being proposed.

 

[rpeters+groups]

Thank you, I believe this is the approach we will take. I will contact
those manufacturers as suggested, and appreciate your help.

 

[Arno Wagner]

Thank you, I believe this is the approach we will take. I will contact
those manufacturers as suggested, and appreciate your help.

You are welcome.

Arno