a million terabytes per day? faaaaaaaaaaaaaaaaaaaaaaaaaark!

N

Nomen Nescio

I read about the SKA (radio telescope) that it will record
900 to 1000 petabytes per day! Much larger than other big science
projects do. I can see 30 TB hard drives within a few years,
what will we have in 2020 when this monster is online?

(I also read that they need some new supercomputer architechture
to process it - the cluster made of boards with 4 Xeons will be
woefully inadequate).
 
R

Rod Speed

Timothy Daniels said:
David Brown wrote
When the expense to gather the raw data is so high, the raw data is
usually saved so that it can be processed in other ways that were not
anticipated when the project began.

Just not feasible with that volume of data.
It is analogous to evidence in a criminal case being kept for decades
after the crime and even after a prosecution.

Nothing like it in fact. It cost peanuts to keep the
evidence with criminal cases. Not so with the SKA.
There is no telling how it might prove important in the future.

Its just not feasible.
Think of how DNA evidence has proven so important for cases whose evidence
was gathered and stored decades before there was even such thing as DNA
analysis.

Sure, but it costs peanuts to keep that evidence.
Think of "Hey! What's that? Did you see that? Run it again, but this
time with the gamma filter."

Its just not possible to keep that volume of data forever.

And that isnt done with the current radio telescopes either.
 
R

Rod Speed

David Brown said:
Timothy Daniels wrote
As Rod says, storing the raw data is infeasible - it won't happen. Even
if it /were/ possible, it still wouldn't happen, for two reasons. First,
it really is /raw/ data - most of it is noise or repetition. They will
filter and analyse it to reduce the quantity and increase the proportion
of information. Secondly, the data is not irreplaceable - baring a few
needle-in-a-haystack events, most of what they get is the same each time
they look at the same part of the sky. So there is no need to keep the
data - thus breaking the analogy with crime evidence.
Also, you've been watching far too much CSI if you think DNA evidence is
that important. Despite the enormous DNA databases some police forces
like to build up, the number of cases where DNA has actually been a major
help in solving a crime is tiny. In the majority of cases, DNA is used
for additional evidence and confirmation (i.e., the police figure out
likely suspects by traditional methods, then use DNA to confirm).

That's because they only have criminals and suspects on the
DNA database, so there is no alternative to that approach.

It is the best way of ruling out particular suspects tho.
 
R

Rod Speed

That's only partly true.

Nope, its completely true. DNA at the crime scene isnt any use
for catching the criminal if the criminal's DNA isnt available to
compare with the DNA that's found at the crime scene. Its only
useful when the more traditional approaches has turned up
suspects and can THEN be used to confirm if there is any of
that suspect's DNA at the crime scene with no plausible reason
for how it got there.
First, some countries (like the UK) are obsessed with keeping as much DNA
as they can - suspects, victims, people "helping the police with their
inquiries", and anyone else the police get hold of will get their DNA in
the database.

That is however a microscopic minority of countrys so whats
possible there isnt relevant to your general claim about DNA

And even there, **** all of the population has their DNA on
any database so **** all crime can be solved by just comparing
the DNA found at the crime scene with the database.
And once there, it is almost impossible to get it removed. The UK DNA
database has over 9M entries - one in seven citizens.

Still **** all of the total population, so **** all crime can be solved by
just comparing the DNA found at the crime scene with the database.
Secondly, crime scenes or victims usually have lots of different DNA.
Say a shop gets robbed. You can take DNA samples from everything in
sight - if you think you know who did it, then you can match the
suspect's DNA with spit found on the counter. But you typically can't
work the other way by looking up all the DNA samples and treating the
owners as suspects.

Yes, but most crime scenes arent like that. Most crime scenes do in
fact have DNA which is much more likely than not to have come from
the criminal and so can be used to confirm that particular suspect does
have his DNA at the crime scene without any plausible explanation
for how it got there.
Thirdly, DNA evidence is often considered as only circumstantial,

That's just plain wrong.
and not even enough to get a search warrant,

That's just plain wrong with crime scene DNA
and with a hit on the criminal DNA database.
as there are many ways your DNA can end up in odd places.

That's just plain wrong with most crime scene DNA.
Sometimes it can help, such as by verifying an alibi, but lack of DNA at
the scene does not mean the suspect was not there.

Yes, but presence of DNA does show that the suspect was there
most of the time.

Just another mindlessly one eyed view...
Some quotations:

Irrelevant to your claim about how DNA is used.

That's just weasel words. There has been a massive increase in
the number of crimes where the guilt of the accused has been
proven using DNA and even more examples of where false
convictions have not happened because DNA proved that
a particular suspect wasn't the criminal that did the crime.

But they did allow them to be dismissed as suspects and so
allowed the police to concentrate on those who did the crime.

Yes, you can certainly make a case that it would be better to
remove those from the database and only retain the DNA of
convicted criminals on the database, but that's an entirely
separate argument to your original claim about DNA.

Nothing random about collecting the DNA of those.
Keeping the DNA of criminals on record is much like keeping their
fingerprints - it's an aid in catching or convicting people a second time.
But that's all.

Wrong. Its also a very effective way of catching the criminals
who do more crime later.
 
R

Rod Speed

Rod Speed said:
Nope, its completely true. DNA at the crime scene isnt any use
for catching the criminal if the criminal's DNA isnt available to
compare with the DNA that's found at the crime scene. Its only
useful when the more traditional approaches has turned up
suspects and can THEN be used to confirm if there is any of
that suspect's DNA at the crime scene with no plausible reason
for how it got there.


That is however a microscopic minority of countrys so whats
possible there isnt relevant to your general claim about DNA

And even there, **** all of the population has their DNA on
any database so **** all crime can be solved by just comparing
the DNA found at the crime scene with the database.


Still **** all of the total population, so **** all crime can be solved by
just comparing the DNA found at the crime scene with the database.


Yes, but most crime scenes arent like that. Most crime scenes do in
fact have DNA which is much more likely than not to have come from
the criminal and so can be used to confirm that particular suspect does
have his DNA at the crime scene without any plausible explanation
for how it got there.


That's just plain wrong.


That's just plain wrong with crime scene DNA
and with a hit on the criminal DNA database.


That's just plain wrong with most crime scene DNA.



Yes, but presence of DNA does show that the suspect was there
most of the time.


Just another mindlessly one eyed view...


Irrelevant to your claim about how DNA is used.


That's just weasel words. There has been a massive increase in
the number of crimes where the guilt of the accused has been
proven using DNA and even more examples of where false
convictions have not happened because DNA proved that
a particular suspect wasn't the criminal that did the crime.


But they did allow them to be dismissed as suspects and so
allowed the police to concentrate on those who did the crime.

Yes, you can certainly make a case that it would be better to
remove those from the database and only retain the DNA of
convicted criminals on the database, but that's an entirely
separate argument to your original claim about DNA.


Nothing random about collecting the DNA of those.


Wrong. Its also a very effective way of catching the criminals
who do more crime later.

And for working out what other crimes criminals who do get caught
have done before they were caught, particularly with cold cases where
the crime scene DNA Is compared with the criminal DNA database a long
time after the crime and is by far the best approach with cold cases.

And can also see falsely convicted criminals released when a hit turns
up in the criminal DNA database long after the false conviction because
the criminal who did the crime gets caught doing a different crime.
 
F

Franc Zabkar

I can see 30 TB hard drives within a few years,
what will we have in 2020 when this monster is online?

http://www.isuppli.com/Memory-and-S...-of-Maximum-Hard-Drive-Densities-by-2016.aspx

"Heat Assisted Magnetic Recording (HAMR) technology is likely to lead
the way in creating next-generation HDDs, even though satisfactory
costs via HAMR comparable to those of PMR have yet to be seen. In
theory, however, advanced technologies like HAMR could extend HDD
areal density to a range spanning 5 to 10 Tb per square inch.

The highest capacity for 3.5-inch HDDs could then reach 30 to 60 TB,
while the smaller and thinner 2.5-inch HDDs used in increasingly
popular thinner notebooks could reach 10 to 20 TB."

The following article discusses HAMR:
http://hddscan.blogspot.com.au/2011/01/hamr-or-path-of-jedi.html

- Franc Zabkar
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Top