Simple Hash algorithm to detect duplicate content

  • Thread starter Thread starter DotNetNewbie
  • Start date Start date
DotNetNewbie said:
My message column is NTEXT(MAX), and it is going to have articles in
it.
I'll look into this approach....
Click to expand...

You mean NTEXT *or* NVARCHAR(MAX) ?

Well - neither can be indexed by SQLServer ...

Arne
 
Jon said:
But that's exactly what an indexed unique constraint would do, but in a
more transparent fashion.

I've only ever had to manually store a hash in a database once, and
that was to effectively hash an unknown-until-execution-time number of
Guids when populating a set of sets.

Databases know how to index text columns. I think it's best to let them
do their job.
Click to expand...

SQLServer does not.

To quote from BOL:

#Columns that are of the large object (LOB) data types ntext, text,
#varchar(max), nvarchar(max), varbinary(max), xml, or image cannot be
#specified as key columns for an index.

Arne
 
Arne Vajhøj said:
SQLServer does not.

To quote from BOL:

#Columns that are of the large object (LOB) data types ntext, text,
#varchar(max), nvarchar(max), varbinary(max), xml, or image cannot be
#specified as key columns for an index.
Click to expand...

That's a pity - and it makes life a bit awkward.

The OP could use a hash and then fetch all values which have the same
hash, then performing the comparison.
 
That's a pity - and it makes life a bit awkward.

The OP could use a hash and then fetch all values which have the same
hash, then performing the comparison.
Click to expand...

Indeed. If you "scroll up" you'll see that this is exactly what I
suggested.
 
Christopher Van Kirk said:
Indeed. If you "scroll up" you'll see that this is exactly what I
suggested.
Click to expand...

True - but before we knew that the database couldn't do it all for the
OP in the first place. Where feasible (i.e. not using NTEXT columns
etc) I'd still favour the unique (indexed) constraint in the DB
approach.
 
Back
Top