USB Flash Drive Integrity Check via "Surface Scan"

[jaugustine]

Hi,

I know that performing a "Surface Scan" on a HD (hard disk) via
"ScanDisk" was useful in finding a bad sector(s), but I wonder if performing a
"Surface Scan" on a USB flash drive will expose an integrity flaw?

Thank You in advance, John

 

[Paul]

Hi,

I know that performing a "Surface Scan" on a HD (hard disk) via
"ScanDisk" was useful in finding a bad sector(s), but I wonder if performing a
"Surface Scan" on a USB flash drive will expose an integrity flaw?

Thank You in advance, John

The USB Flash should have much the same facilities as a hard drive.

On a hard drive, a bad sector is spared out automatically.
When there are no longer spares available near the bad sector,
the sector is then "really bad". An attempt to read the bad
sector then, returns a CRC error. And the OS may then get to
add the bad cluster, to a bad cluster list.

So surface scanning the hard drive, doesn't necessarily expose the
"problem" in a useful way. Only when it's almost too late, does
your first bad cluster show up. Using S.M.A.R.T statistics and
reallocated sectors, you get an earlier indication, but even
then, I suspect up to a certain level of usage of spares,
the stats will show nothing. (If the scale was perfectly linear,
users could "cherry pick" hard drives, and send back any drive
showing even one error. By preventing the stats from showing
low numbers of errors, the user cannot figure out whether to
keep the drive or send it back.)

A USB flash, could have sparing at the block level. And an error
detecting code, can tell when a block has correctable errors,
or so many errors as to be uncorrectable. Unfortunately, the
documentation for flash controllers and chips, isn't good enough
to develop a more detailed picture of how it all works.

I don't think a USB flash has S.M.A.R.T, so you don't have that
to work with either.

If you surface scan the USB flash, it's going to look "perfect",
up until the day it runs out of spare blocks, and in all probability,
it'll die the same day. So it's not like there would be a long
interval between "spotting trouble" and "it's dead Jim".

You would need a utility that can "reach the flash chips"
and talk to them directly. And get info about how many blocks
are currently spared out. The manufacturer may have such a
utility, but they're not generally shared with the public.

One indicator of health, might be access speed, but I don't
have any useful information on how to relate the two. I feel
a flash stick that is slowing down, isn't healthy, but I can't
tell you how many days are left if it is running at about half
speed.

I guess this is why I would recommend a USB flash for "data transport"
but not for "archival storage". Never put your only copy of some
data, on USB flash.

Paul

 

[Paul]

Somewhat relatedly, I wonder what media is reliable after a long time (like
20 years or so)?

I've had some CD's fail, and we all know HDs fail, and I would expect DVS's
fail too, so I guess that leaves tape??? Although even that may be
problematic. Maybe there isn't anything that's reliable for the long haul.

There are companies making media specifically for archival storage.
And apparently, that media doesn't fit in a CD/DVD burner either.
And, it has a price to match.

None of the regular stuff, would I trust to just leave it in a
cardboard box until I needed it. Transfer to other media on a
semi-regular basis, is probably the best plan.

I class a hard drive as "archival storage" in the near term.
Whereas, I would never leave my only copy of valuable data on
a USB flash, even for a few days. Enough people have had
those fail, I wouldn't take the chance. If you need to take
files from home to work, you'd leave the copy on the home
machine, until you were sure the copy made it to work.

When you go to a big box store, and look at the USB flash for
sale, it's mainly miserable 4MB/sec stuff. It doesn't inspire
confidence, when you bench it. It's crap that didn't make it
into other products (floor sweepings). I've got only one
good USB flash here, that writes at about 17MB/sec and reads
at 30MB/sec, and the rest are pretty slow by comparison.
I have Linux loaded on a slow one, and it must take about
4 minutes to boot. I think using a boot CD would be faster
(maybe 3 minutes).

Paul

 

[Bob Willard]

Somewhat relatedly, I wonder what media is reliable after a long time (like
20 years or so)?

I've had some CD's fail, and we all know HDs fail, and I would expect DVS's
fail too, so I guess that leaves tape??? Although even that may be
problematic. Maybe there isn't anything that's reliable for the long haul.

Paper. Good old paper, printed with decent ink, will last for centuries
if properly stored.

Paper is better than any magnetic media, unless you include some process
to periodically read from old and record onto current media. Even then,
paper has a track record which is better than magnetic stuff by orders
of magnitude.

Alternatively, rock inscribed with chisel may be even more stable than
paper, but the writing rate sucks.

 

[Bob Willard]

The data density isn't that great, either. Punched plastic tape is
better, but still pretty low density! (Although it seems such a robust
medium, I don't think I _have_ ever seen it used for archival storage;
its application seems to be, or did, use for machine control in hostile
environments, like a dirty workshop.)

When I last used punched tape, we used metalized mylar to ship programs
and data to rather humid sites (think outdoors 10° or so north of the
equator).

Our data center had high-speed tape punches: 63.3 B/s, which is almost
2 GB/year; but punching metalized mylar would destroy a tape punch long
before the first GB was punched.

 

[glee]

Bill in Co wrote: []

Somewhat relatedly, I wonder what media is reliable after a long
time (like 20 years or so)?
I've had some CD's fail, and we all know HDs fail, and I would

I think CDs're more reliable if you write them at a slow speed, too.

expect DVS's fail too, so I guess that leaves tape??? Although
even that may be problematic. Maybe there isn't anything that's
reliable for the long haul.

[]
None of the regular stuff, would I trust to just leave it in a
cardboard box until I needed it. Transfer to other media on a
semi-regular basis, is probably the best plan.

Apart from anything else, the reader hardware may deteriorate too (and
not be available to replace).

I class a hard drive as "archival storage" in the near term.
Whereas, I would never leave my only copy of valuable data on
a USB flash, even for a few days. Enough people have had
those fail, I wouldn't take the chance. If you need to take
files from home to work, you'd leave the copy on the home
machine, until you were sure the copy made it to work.

When you go to a big box store, and look at the USB flash for
sale, it's mainly miserable 4MB/sec stuff. It doesn't inspire
confidence, when you bench it. It's crap that didn't make it
into other products (floor sweepings). I've got only one
good USB flash here, that writes at about 17MB/sec and reads
at 30MB/sec, and the rest are pretty slow by comparison.
I have Linux loaded on a slow one, and it must take about
4 minutes to boot. I think using a boot CD would be faster
(maybe 3 minutes).

My main concern about flash - USB sticks or SSDs - is a fear (I don't
know how well-founded; just based on personal experience) that they'll
fail suddenly and totally, whereas HDs have usually, in my experience,
given warning. I know this is not something you should rely on.

"Home-burned' optical discs deteriorate over time... get "holes" in the
data burned to the disc.

My experience with flash memory (flash drives and SSDs) is the same...
failure without warning. SSD failure is usually catastrophic... hard
drives can also fail catastrophically but usually don't. Flash memory
supposedly has a shorter life cycle than hard drive magnetic storage
(barring mechanical failure of the hard drive)... flash cells have a
limited life span, a block on an SSD can only be erased a limited number
of times. Very small-capacity SSD drives can fail in a relatively short
time if data is erased and written over a large portion of the SSD
repeatedly... which is what an OS does. The tiny SSDs in some netbooks
have a history of poor life span.

 

[Paul]

glee wrote:

My experience with flash memory (flash drives and SSDs) is the same...
failure without warning. SSD failure is usually catastrophic... hard
drives can also fail catastrophically but usually don't. Flash memory
supposedly has a shorter life cycle than hard drive magnetic storage
(barring mechanical failure of the hard drive)... flash cells have a
limited life span, a block on an SSD can only be erased a limited number
of times. Very small-capacity SSD drives can fail in a relatively short
time if data is erased and written over a large portion of the SSD
repeatedly... which is what an OS does. The tiny SSDs in some netbooks
have a history of poor life span.

I've not seen much in the way of post-mortem data for SSDs,
but based on how long the user owned them, they seem to die
before the write cycles are exhausted. Which implies a problem
with data structures, rather than just the ability to store
raw data. They can do a fair amount of moving around of
data, when you're not using them. (Like, if you abuse them
with some 4KB random write benchmarks.)

Some SSDs have a "supercap", which provides power when
the power goes off. That allows outstanding writes to finish.
That's a feature of enterprise drives. It's not clear to
what extent that kind of thing is a problem for consumer
drives. If you have a UPS, you might already have
an orderly shutdown process in place anyway. (The concern
there would be, a data structure write inside the SSD,
doesn't get finished properly.) I suppose it all depends,
on how "atomic" the handling of structures is within the
SSD (atomic commit).

http://en.wikipedia.org/wiki/Atomic_commit

Paul

 

[Bob F]

Paper. Good old paper, printed with decent ink, will last for
centuries if properly stored.

Paper is better than any magnetic media, unless you include some
process to periodically read from old and record onto current media. Even
then, paper has a track record which is better than magnetic
stuff by orders of magnitude.

Alternatively, rock inscribed with chisel may be even more stable than
paper, but the writing rate sucks.

Or, try these drives and disks.

http://www.freesoftwaremagazine.com...ives_millenniatas_new_archival_dvd_technology