calculate an unique id of a file

[timor.super]

Hi group,

I need a way of calculating an unique id for a file.

I've seen things like Crc32, 64, checksum .... there's a list here :
http://en.wikipedia.org/wiki/List_of_hash_functions

What is the best option for me ? I have to identify larges files and
small files. I need something fast if possible. How can I be sure that
it is unique ?

Thanks for any advice

 

[Hans Kesting]

(e-mail address removed) pretended :

Hi group,

I need a way of calculating an unique id for a file.

I've seen things like Crc32, 64, checksum .... there's a list here :
http://en.wikipedia.org/wiki/List_of_hash_functions

What is the best option for me ? I have to identify larges files and
small files. I need something fast if possible. How can I be sure that
it is unique ?

Thanks for any advice

I don't think you can guarantee that you can identify file uniquely by
some hash code.
Say you calculate a hash of a single byte. This can hold 256 distinct
values, so by the time you encode file #257 you *will* have found a
duplicate hashcode. For a hash of a 32-bit integer, that amount is in
the billions while larger sizes go to astronomical numbers, but still
you cannot guarantee that there will be no duplicates.

As for speed, large files will require more time, as every byte has to
be read and processed.

Hans Kesting

 

[Peter Morris]

Based only on the data or the filename too?

If you have the same filename, same size, and same hashcode it is likely it
is the same file. However, you don't say whether or not the filename is
important. If it isn't a factor then I would probably make the ID based on

FileSize/Hash1/Hash2

Where Hash1 and Hash2 are the results of two different hash algorithms.


Pete
====
http://mrpmorris.blogspot.com
http://www.capableobjects.com

 

[timor.super]

No hash function can guarantee uniqueness; a CRC32 will have a
collision probability of between 1 and 1 in 2^32.  The 1 is for cases
where you should have used a cryptographically secure hash function,
i.e. where there is someone deliberately trying to break your system
or the Data Protection law requires a reasonable level of security.

For non-cryptographic use go for a CRC with sufficient size to reduce
the collision probability to an acceptably low level.

For cryptographic purposes use SHA-256 or SHA-512.  The MD series are
broken and SHA-384 just calculates a SHA-512 result and truncates it
so you might as well go for SHA-512.

rossum

Thanks for your answer,
I think I don't need SHA.

In fact, I have to know if a file has been modified between to access.
Then, if I use a crc64, it should be enough to know that file has been
modified. Ins't it ?

Do you think this class is good ? http://damieng.com/blog/2007/11/19/calculating-crc-64-in-c-and-net

Best regards

 

[Arne Vajhøj]

I need a way of calculating an unique id for a file.

I've seen things like Crc32, 64, checksum .... there's a list here :
http://en.wikipedia.org/wiki/List_of_hash_functions

What is the best option for me ? I have to identify larges files and
small files. I need something fast if possible. How can I be sure that
it is unique ?

Any CRC or Hash should do.

Hash will have less probability of collisions than CRC.

If you need to guarantee no collisions, then you can not use
any of them.

If you can live with a small probability of collisions, then
everything is possible.

The important questions is then whether you need to worry about
files with identical start but different end. Because if not then
you could speed up the process a lot by only checksumming
up to like 100 KB of data.

Arne