I have a project that would feed a continual stream of data at a rate of
approximately 17 GB per day at a minimum. My project would require archiving
this data indefinately. I would like to know of the cheapest possible
solution to store 17 GB per day for as long of a time span as possible.

I would highly prefer to have a medium with an extremely long shelf life.
That is, I do not want to find myself 10 years from now trying to copy 62
Terrabytes of data off of piles and piles of old storage discs/cartridges
onto a new medium to preserve the data. I want it to last as long as
possible from the get go.

Another important thing to consider is the schedule of when discs/cartridges
would have to be replaced. I will not be able to physically replace storage
media any more frequently than once per 24 hour cycle. That is, the
individual discs or cartridges must store a minimum of 17 GB each. I would
highly prefer this to be on a 48 hour cycle or more, as I might not be
around every single day. That is, I'd prefer to have 34 GB or more per
disc/cartridge.

I imagine I will have multiple physical drives setup in such a way that as
soon as one drive is full, the server begins saving information to the other
drive(s). This way when I arrive to change media, I can change all of the
drives that are full while the system is still writeing data to the one
active drive. This could help offset the need for larger capacities per
unit, as I could have multiple units running live at any given moment.

I realize this is going to cost quite a bit to accomplish over the next
decade or two. However, I would like to keep the costs down as low as
possible. This is really just a hobby to me, so anything that must be
purchased will come soley out of my pocket with no commercial backing.

What is the cheapest possible way to archive 17 GB per day and save it for
decades to come? Would magnetic tape have too short of a shelf life? Would
optical discs be too expensive?


Thanks for your help,

-Jeff

On Mon, 24 Nov 2003 10:29:34 -0500
"Jeff" <jeff_phil-(E-Mail Removed)> wrote:

> I have a project that would feed a continual stream of data at a rate
> of approximately 17 GB per day at a minimum. My project would require
> archiving this data indefinately. I would like to know of the cheapest
> possible solution to store 17 GB per day for as long of a time span as
> possible.

Which do you want, do you want the cheapest solution or the one that
holds the data for the longest possible span?

> I would highly prefer to have a medium with an extremely long shelf
> life. That is, I do not want to find myself 10 years from now trying
> to copy 62 Terrabytes of data off of piles and piles of old storage
> discs/cartridges onto a new medium to preserve the data. I want it to
> last as long as possible from the get go.

In that case you're talking CD or magneto-optical or possibly DVD.

> Another important thing to consider is the schedule of when
> discs/cartridges would have to be replaced. I will not be able to
> physically replace storage media any more frequently than once per 24
> hour cycle. That is, the individual discs or cartridges must store a
> minimum of 17 GB each.

If you have to have 17 GB minimum capacity then your best bet is
probably DLT, which is good for 10-20 years.

> I would highly prefer this to be on a 48 hour
> cycle or more, as I might not be around every single day. That is, I'd
> prefer to have 34 GB or more per disc/cartridge.

DLT will handle that nicely.

> I imagine I will have multiple physical drives setup in such a way
> that as soon as one drive is full, the server begins saving
> information to the other drive(s).

In that case why do you need 17 GB on each cartridge?

> This way when I arrive to change
> media, I can change all of the drives that are full while the system
> is still writeing data to the one active drive. This could help offset
> the need for larger capacities per unit, as I could have multiple
> units running live at any given moment.
>
> I realize this is going to cost quite a bit to accomplish over the
> next decade or two. However, I would like to keep the costs down as
> low as possible. This is really just a hobby to me, so anything that
> must be purchased will come soley out of my pocket with no commercial
> backing.
>
> What is the cheapest possible way to archive 17 GB per day and save it
> for decades to come? Would magnetic tape have too short of a shelf
> life? Would optical discs be too expensive?

You have a tradeoff to make, and only you can know how much you are
willing to spend for what capability.

> Thanks for your help,
>
> -Jeff
>
>


--
--
--John
Reply to jclarke at ae tee tee global dot net
(was jclarke at eye bee em dot net)

Previously Jeff <jeff_phil-(E-Mail Removed)> wrote:
> I have a project that would feed a continual stream of data at a rate of
> approximately 17 GB per day at a minimum. My project would require archiving
> this data indefinately. I would like to know of the cheapest possible
> solution to store 17 GB per day for as long of a time span as possible.

> I would highly prefer to have a medium with an extremely long shelf life.
> That is, I do not want to find myself 10 years from now trying to copy 62
> Terrabytes of data off of piles and piles of old storage discs/cartridges
> onto a new medium to preserve the data. I want it to last as long as
> possible from the get go.

> Another important thing to consider is the schedule of when discs/cartridges
> would have to be replaced. I will not be able to physically replace storage
> media any more frequently than once per 24 hour cycle. That is, the
> individual discs or cartridges must store a minimum of 17 GB each. I would
> highly prefer this to be on a 48 hour cycle or more, as I might not be
> around every single day. That is, I'd prefer to have 34 GB or more per
> disc/cartridge.

> I imagine I will have multiple physical drives setup in such a way that as
> soon as one drive is full, the server begins saving information to the other
> drive(s). This way when I arrive to change media, I can change all of the
> drives that are full while the system is still writeing data to the one
> active drive. This could help offset the need for larger capacities per
> unit, as I could have multiple units running live at any given moment.

> I realize this is going to cost quite a bit to accomplish over the next
> decade or two. However, I would like to keep the costs down as low as
> possible. This is really just a hobby to me, so anything that must be
> purchased will come soley out of my pocket with no commercial backing.

> What is the cheapest possible way to archive 17 GB per day and save it for
> decades to come? Would magnetic tape have too short of a shelf life? Would
> optical discs be too expensive?

For your requirements you need an MOD storage libary or multiple MOD
drives. Either with 2.3GB 3.5" disks or 5.25" disks. This is the
only solution that will give you the shelf-life you want. Others claim
to reach it, but fail. CD-R/DVD-R will likely last several years but
not several decades. Tapes of any sort get problematic after a decade.
Same for HDDs. In addition MOD is the only medium type where the
manufacturers are comitted to long term drive availability and
backwards compatibility, since it was explicitely designed for
professional long-term storage without a special storage
environment.

One option would be to use 10 conventional 3.5" MOD drives with
some software that switches to the next disk when one is full.
This would cost about 400 Euro per USB drive and and around 25 Euro
per disk. Say 4000 Euro basic investment plus 200 Euro/day.

If you can spend more money, you could go for an optical jukebox
like this:

http://www.promarktech.com/storage/mo/plasmon/m104.htm

Gives you up to 55 days with your storage needs before you
have to replace the cartridges. The basic equipment will be a little
expensive. A 5.4GB disk is around 55 Euro, so you can expect an
additional cost of around 200 Euro/day, just like in the above case.
I could not find a price quote for 9GB disks.

If you cannot spend these 200 Euro/day, then forget it. You will
find ceaper solutions that may keep half a decade or maybe even a
decade, but not longer.

Arno
--
For email address: lastname AT tik DOT ee DOT ethz DOT ch
GnuPG: ID:1E25338F FP:0C30 5782 9D93 F785 E79C 0296 797F 6B50 1E25 338F
"The more corrupt the state, the more numerous the laws" - Tacitus

I don't have an answer, but I do know that whatever media device you choose
will be obsolete in far too few years. For long term storage, you need
durable media - but you will also need to have on hand the media device
(tape drive, DVD drive, or whatever) because there is a very good
possibility that such a device will be unavailable a few years from now.
Computer technology is a rapidly evolving entity.

I think that most who have been involved in archiving find that copying data
from an older medium to a more modern one is a necessary and periodic
process. In other words, do not just store the data now and expect to be
able to retrieve it 10 to 15 years down the road. It probably should be
transferred to newer media much more frequently than that.

My opinion. Yours may differ.

"Jeff" <jeff_phil-(E-Mail Removed)> wrote in message
news:(E-Mail Removed)...
> I have a project that would feed a continual stream of data at a rate of
> approximately 17 GB per day at a minimum. My project would require
archiving
> this data indefinately. I would like to know of the cheapest possible
> solution to store 17 GB per day for as long of a time span as possible.
>
> I would highly prefer to have a medium with an extremely long shelf life.
> That is, I do not want to find myself 10 years from now trying to copy 62
> Terrabytes of data off of piles and piles of old storage discs/cartridges
> onto a new medium to preserve the data. I want it to last as long as
> possible from the get go.
>
> Another important thing to consider is the schedule of when
discs/cartridges
> would have to be replaced. I will not be able to physically replace
storage
> media any more frequently than once per 24 hour cycle. That is, the
> individual discs or cartridges must store a minimum of 17 GB each. I would
> highly prefer this to be on a 48 hour cycle or more, as I might not be
> around every single day. That is, I'd prefer to have 34 GB or more per
> disc/cartridge.
>
> I imagine I will have multiple physical drives setup in such a way that as
> soon as one drive is full, the server begins saving information to the
other
> drive(s). This way when I arrive to change media, I can change all of the
> drives that are full while the system is still writeing data to the one
> active drive. This could help offset the need for larger capacities per
> unit, as I could have multiple units running live at any given moment.
>
> I realize this is going to cost quite a bit to accomplish over the next
> decade or two. However, I would like to keep the costs down as low as
> possible. This is really just a hobby to me, so anything that must be
> purchased will come soley out of my pocket with no commercial backing.
>
> What is the cheapest possible way to archive 17 GB per day and save it for
> decades to come? Would magnetic tape have too short of a shelf life? Would
> optical discs be too expensive?
>
>
> Thanks for your help,
>
> -Jeff
>
>

"Papa" wrote:
> [..........]
> I think that most who have been involved in archiving find that copying data
> from an older medium to a more modern one is a necessary and periodic
> process. In other words, do not just store the data now and expect to be
> able to retrieve it 10 to 15 years down the road. It probably should be
> transferred to newer media much more frequently than that.


True. When I left a job 12 years ago, I archived lots of my work on
5 1/4 in. floppies. I recently had a reason to access some of that stuff,
and guess what - no one I know has a 5 1/4 in. drive, much less one
that will run on Windows.

*TimDaniels*

In article <(E-Mail Removed)>, Jeff wrote:

> I have a project that would feed a continual stream of data at a rate of
> approximately 17 GB per day at a minimum. My project would require archiving
> this data indefinately. I would like to know of the cheapest possible
> solution to store 17 GB per day for as long of a time span as possible.

I saw a 400 DVD changer drive at Fry's Electronics the other day.
It was selling for "just" US$1600. One places the 400 DVDs vertically
within a circular carosel (a torus). I have no idea what sort of software
is used to drive it though. I wonder if they make a +- BURNER version.

- Dan

"Timothy Daniels" <(E-Mail Removed)> wrote in message
news:NYGdnfKYyfHRH1-iRVn-(E-Mail Removed)...
>
> "Papa" wrote:
> > [..........]
> > I think that most who have been involved in archiving find that copying
data
> > from an older medium to a more modern one is a necessary and periodic
> > process. In other words, do not just store the data now and expect to be
> > able to retrieve it 10 to 15 years down the road. It probably should be
> > transferred to newer media much more frequently than that.
>
>
> True. When I left a job 12 years ago, I archived lots of my work on
> 5 1/4 in. floppies. I recently had a reason to access some of that stuff,
> and guess what - no one I know has a 5 1/4 in. drive, much less one
> that will run on Windows.
>
> *TimDaniels*


Very true indeed. Thanks for pointing that out. I guess I won't have any
possible way of maintaining a storage media for over 10 years or so. So I
will have to be in a continual process of converting old data over to newer
storage mediums every few years. The biggest problem I see with this is that
it would be very time consuming if there were numerous cartridges
(disks/whatever). What that means to me is that I need the individual units
to be able to store massive amounts of data. If the cartridges were 17 GB
each, I would have to replace them every day. If I then had to convert them
to a newer media format 10 years down the road, that would be 3650
cartridges/disks/whatever to swap in and out of drives. That would be way
too many to be able to practically convert to a new format. It would be a
lot easier if I had a storage medium that would last a week instead of a day
(say 120 GB instead of 17 GB) on each cartridge / disk. Then I'd only have
520 disks to convert 10 years down the road.


Thanks,

Jeff

PS: The irony in this however is that I happen to have several 5.25" disk
drives sitting around in my basement, and yes they do run in Windows. None
of them are in a computer that is actually running, but it would probably
only take me 10 minutes or so to toss one in if I needed it. Of course, if
you're talking about disks you made with an Apple or Commadore or some other
funky computer, I have no idea what I'd do with them.

<(E-Mail Removed)> wrote in message
news:(E-Mail Removed)...
> In article <(E-Mail Removed)>, Jeff wrote:
>
> I saw a 400 DVD changer drive at Fry's Electronics the other day.
> It was selling for "just" US$1600. One places the 400 DVDs vertically
> within a circular carosel (a torus). I have no idea what sort of software
> is used to drive it though. I wonder if they make a +- BURNER version.
>
> - Dan

I really like that idea! I could just pile on the disks once in a while and
not have to keep a close eye on it all the time. The only thing is, a DVD
only holds about 4.7 GB. I'll be needing around 17 GB per day, so that would
go through about 25 DVDs a week. After 10 years I'd have 13 thousand DVDs to
keep up with. That'd be quite a collection. I think I'll keep looking for
something that can hold more data on each disk/cartridge/unit/whatever.

Thanks though,

-Jeff

"Jeff" <jeff_phil-(E-Mail Removed)> wrote in message news:(E-Mail Removed)...
> "Timothy Daniels" <(E-Mail Removed)> wrote in message
> news:NYGdnfKYyfHRH1-iRVn-(E-Mail Removed)...
> >
> > "Papa" wrote:
> > > [..........]
> > > I think that most who have been involved in archiving find that copying
> data
> > > from an older medium to a more modern one is a necessary and periodic
> > > process. In other words, do not just store the data now and expect to be
> > > able to retrieve it 10 to 15 years down the road. It probably should be
> > > transferred to newer media much more frequently than that.
> >
> >
> > True. When I left a job 12 years ago, I archived lots of my work on
> > 5 1/4 in. floppies. I recently had a reason to access some of that stuff,
> > and guess what - no one I know has a 5 1/4 in. drive, much less one
> > that will run on Windows.

> Very true indeed. Thanks for pointing that out. I guess I won't have any
> possible way of maintaining a storage media for over 10 years or so.

Dunno. That drive availability should be fine for CDs and DVDs.

The media lasting that long is more of a dubious proposition with DVDs currently.

> So I will have to be in a continual process of converting
> old data over to newer storage mediums every few years.

What the hell is that volume, 17GB a day ?

> The biggest problem I see with this is that it would be very time
> consuming if there were numerous cartridges (disks/whatever).

Yeah, you'd certainly need an automatic changer, both with the
old and the new. Likely DVD would be viable there, at a price.

> What that means to me is that I need the individual units to
> be able to store massive amounts of data. If the cartridges
> were 17 GB each, I would have to replace them every day.

How crucial is that data if you say lose 1% of them ?

> If I then had to convert them to a newer media
> format 10 years down the road, that would be 3650
> cartridges/disks/whatever to swap in and out of drives.

Yeah, tho changers are buyable, at a price.

> That would be way too many to be able
> to practically convert to a new format.

Not with a changer.

> It would be a lot easier if I had a storage medium
> that would last a week instead of a day (say 120
> GB instead of 17 GB) on each cartridge / disk.

Yeah, and starting to be viable to use
hard drives, a new one every week etc.

Dunno that I'd like to risk them lasting 10 years tho, even
if stored ideally. Probably viable, but you could end up with
spectacular problems if say the bearings gum up over that time.

> Then I'd only have 520 disks to convert 10 years down the road.

Yeah, might be viable with hard drive I spose. Tho you'd
really have to duplicate them if you cant risk any losses at all.

> PS: The irony in this however is that I happen to have several 5.25" disk
> drives sitting around in my basement, and yes they do run in Windows.

Yeah, me too.

> None of them are in a computer that is actually running, but it would
> probably only take me 10 minutes or so to toss one in if I needed it.

Yeah, very likely.

> Of course, if you're talking about disks you made
> with an Apple or Commadore or some other funky
> computer, I have no idea what I'd do with them.

You could use them as frizbees.

On Mon, 24 Nov 2003 14:12:26 -0800, "Timothy Daniels"
<(E-Mail Removed)> wrote:

>
>"Papa" wrote:
>> [..........]
>> I think that most who have been involved in archiving find that copying data
>> from an older medium to a more modern one is a necessary and periodic
>> process. In other words, do not just store the data now and expect to be
>> able to retrieve it 10 to 15 years down the road. It probably should be
>> transferred to newer media much more frequently than that.
>
>
> True. When I left a job 12 years ago, I archived lots of my work on
>5 1/4 in. floppies. I recently had a reason to access some of that stuff,
>and guess what - no one I know has a 5 1/4 in. drive, much less one
>that will run on Windows.
>
>*TimDaniels*

I kept mine

it's in the closet.