Hi group,

I need a way of calculating an unique id for a file.

I've seen things like Crc32, 64, checksum .... there's a list here :
http://en.wikipedia.org/wiki/List_of_hash_functions

What is the best option for me ? I have to identify larges files and
small files. I need something fast if possible. How can I be sure that
it is unique ?

Thanks for any advice

(E-Mail Removed) pretended :
> Hi group,
>
> I need a way of calculating an unique id for a file.
>
> I've seen things like Crc32, 64, checksum .... there's a list here :
> http://en.wikipedia.org/wiki/List_of_hash_functions
>
> What is the best option for me ? I have to identify larges files and
> small files. I need something fast if possible. How can I be sure that
> it is unique ?
>
> Thanks for any advice

I don't think you can guarantee that you can identify file uniquely by
some hash code.
Say you calculate a hash of a single byte. This can hold 256 distinct
values, so by the time you encode file #257 you *will* have found a
duplicate hashcode. For a hash of a 32-bit integer, that amount is in
the billions while larger sizes go to astronomical numbers, but still
you cannot guarantee that there will be no duplicates.

As for speed, large files will require more time, as every byte has to
be read and processed.

Hans Kesting

Based only on the data or the filename too?

If you have the same filename, same size, and same hashcode it is likely it
is the same file. However, you don't say whether or not the filename is
important. If it isn't a factor then I would probably make the ID based on

FileSize/Hash1/Hash2

Where Hash1 and Hash2 are the results of two different hash algorithms.


Pete
====
http://mrpmorris.blogspot.com
http://www.capableobjects.com

On 17 oct, 19:18, rossum <rossu...@coldmail.com> wrote:
> On Fri, 17 Oct 2008 05:23:05 -0700 (PDT), timor.su...@gmail.com wrote:
> >Hi group,
>
> >I need a way of calculating an unique id for a file.
>
> >I've seen things like Crc32, 64, checksum .... there's a list here :
> >http://en.wikipedia.org/wiki/List_of_hash_functions
>
> >What is the best option for me ? I have to identify larges files and
> >small files. I need something fast if possible. How can I be sure that
> >it is unique ?
>
> >Thanks for any advice
>
> No hash function can guarantee uniqueness; a CRC32 will have a
> collision probability of between 1 and 1 in 2^32. *The 1 is for cases
> where you should have used a cryptographically secure hash function,
> i.e. where there is someone deliberately trying to break your system
> or the Data Protection law requires a reasonable level of security.
>
> For non-cryptographic use go for a CRC with sufficient size to reduce
> the collision probability to an acceptably low level.
>
> For cryptographic purposes use SHA-256 or SHA-512. *The MD series are
> broken and SHA-384 just calculates a SHA-512 result and truncates it
> so you might as well go for SHA-512.
>
> rossum

Thanks for your answer,
I think I don't need SHA.

In fact, I have to know if a file has been modified between to access.
Then, if I use a crc64, it should be enough to know that file has been
modified. Ins't it ?

Do you think this class is good ? http://damieng.com/blog/2007/11/19/c...4-in-c-and-net

Best regards

(E-Mail Removed) wrote:
> I need a way of calculating an unique id for a file.
>
> I've seen things like Crc32, 64, checksum .... there's a list here :
> http://en.wikipedia.org/wiki/List_of_hash_functions
>
> What is the best option for me ? I have to identify larges files and
> small files. I need something fast if possible. How can I be sure that
> it is unique ?

Any CRC or Hash should do.

Hash will have less probability of collisions than CRC.

If you need to guarantee no collisions, then you can not use
any of them.

If you can live with a small probability of collisions, then
everything is possible.

The important questions is then whether you need to worry about
files with identical start but different end. Because if not then
you could speed up the process a lot by only checksumming
up to like 100 KB of data.

Arne