what is best algorithm to check duplicated rows

[Ryan Liu]

Hi,

If I have tens of thousands DataRow in a DataTable and allow the end user to
pick any DataColumn(s) to check for duplicated lines, the data is so large,
is there a better API, algorithm can be used for this purpose?

Thanks a lot!
Ryan

 

[JR]

SELECT?

JR

 

[Michael Nemtsev]

Hello Ryan,

Hmm,
I see two ways - using the hashtable or sorting + binary search

---
WBR, Michael Nemtsev [.NET/C# MVP].
My blog: http://spaces.live.com/laflour
Team blog: http://devkids.blogspot.com/

"The greatest danger for most of us is not that our aim is too high and we
miss it, but that it is too low and we reach it" (c) Michelangelo

RL> Hi,
RL>
RL> If I have tens of thousands DataRow in a DataTable and allow the end
RL> user to pick any DataColumn(s) to check for duplicated lines, the
RL> data is so large, is there a better API, algorithm can be used for
RL> this purpose?
RL>
RL> Thanks a lot!
RL> Ryan

 

[Guest]

Ryan,
The first question I would ask is "how did you get tens of thousands of
rows" into this Datatable? If they came out of a database, shouldn't that be
where you are enforcing your referential and unique column integrity?
Peter

 

[Ryan Liu]

Hi Peter,



The data is imported by end user from external file, most time it is a csv
text file.



After import to datatable, then the end user specify the criteria which is
used to pick rows from datatable.



Then for selected rows, the end user want to check duplicated lines before
insert them to database. The criteria for checking duplicated lines is
also specified by the end user.



And I am also required check duplicated entries against data already in
database.



Thank you and everyone replyed to this message!

Ryan

 

[Ryan Liu]

Thanks!

Just hope hash algorithm for string is as efficient as int.

And the criteria for checking duplicated datarows could be based on multiple
dataColumns (AND logic), this make it difficult for me to come out a hash
algorithm.

Ryan

 

[Guest]

You can try adding a unique constraint to one or more columns in the DataTable.
I'm not sure exactly how to treat exceptions or error messages during an
import, but I'm sure if you look it up in the MSDN documentation you can find
some examples.
Peter

--
Site: http://www.eggheadcafe.com
UnBlog: http://petesbloggerama.blogspot.com
Short urls & more: http://ittyurl.net

 

[louistoyama]

Assumed "DN" is the column you want to check for duplication

DataTable data = ???;

string duplicatedNumber = string.Empty;

DataRow[] dr = data.Select("DN<>'' AND DN IS NOT NULL", "DN");

string prevKey = string.Empty;

for (int row = 0; row < dr.Length; row++)
{
if (prevKey.Equals(dr[row]["DN"].ToString()) && duplicatedNumber.IndexOf(dr[row]["DN"].ToString()) < 0)
duplicatedNumber += string.Format("{0}<br>", dr[row]["DN"].ToString());
else prevKey = dr[row]["DN"].ToString();
}

I have 12000+ rows and it takes a second to locate the duplication