Avoiding duplicate lines?

[google_groups3]

Hi all.

I currently have 2 text files which contain lists of file names. These
text files are updated by my code. What I want to do is be able to
merge these text files discarding the duplicates.

And to make it harder (or not???!!) my criteria for defining the
duplicate is the left 15 (or so) characters of the file path.


Help, as always, is greatly appreciated. And if you was here with me,
I would buy you a drink to thank you!

Thanks

 

[Bob Hollness]

Hi all.

I currently have 2 text files which contain lists of file names. These
text files are updated by my code. What I want to do is be able to
merge these text files discarding the duplicates.

And to make it harder (or not???!!) my criteria for defining the
duplicate is the left 15 (or so) characters of the file path.


Help, as always, is greatly appreciated. And if you was here with me,
I would buy you a drink to thank you!

Thanks

And duplicate posts!!!! ;-)

 

[Mark Jones]

Not elegant but effective -

You could read each file in turn, line by line into a string.
Put the string into an array only after failing a .binarysearch for the string value.
Then, stream the array back out to a new file.


Read each one in turn into a dataset using an iostream

 

[Cor Ligthert]

Hi,

In my opinion is a simple use of the hashtable all you need.

http://msdn.microsoft.com/library/d...frlrfSystemCollectionsHashtableClassTopic.asp


I hope this helps?

Cor

 

[Bob Hollness]

Thanks Cor. This looks a little too complicated and using too many CPU
cycles. I guess i'll loop each line through the each line of the other
file.

 

[Lucas Tam]

Thanks Cor. This looks a little too complicated and using too many CPU
cycles. I guess i'll loop each line through the each line of the other
file.

Cor's suggestion of a hashtable (or even mine of a datatable) is a good
idea - it's probably using MUCH less CPU cycles than you looping through
the files.

 

[Bob Hollness]

Cor's suggestion of a hashtable (or even mine of a datatable) is a good
idea - it's probably using MUCH less CPU cycles than you looping through
the files.

Really? It just looks like so much more code. (Of course, currently, I
have no idea what a hashtable is.....!)

 

[Lucas Tam]

Really? It just looks like so much more code. (Of course, currently, I
have no idea what a hashtable is.....!)

More code doesn't mean less efficent!

Did you even take a moment to look at my example or Cor's example?

Cor's example is TWO lines long:

Dim Filenames As New Hashtable()
Filenames .Add("Line1", nothing)

I don't think it gets easier than that.

 

[Bob Hollness]

Hi all.

I currently have 2 text files which contain lists of file names. These
text files are updated by my code. What I want to do is be able to
merge these text files discarding the duplicates.

And to make it harder (or not???!!) my criteria for defining the
duplicate is the left 15 (or so) characters of the file path.


Help, as always, is greatly appreciated. And if you was here with me,
I would buy you a drink to thank you!

Thanks

OK. This is the solution I came up with. Not as elegant as one would have
hoped. but then again, only I get to see how it functions under the bonnet
(hood for the Americans) !!! And of course, this is still to be tidied up
and made pretty. Feel free to pull it apart and embarrass me.......


Sub FindDupes(ByVal File2Compare As String, ByVal OriginalFile As
String, ByVal OutputFile As String)

Dim File1Reader As New StreamReader(File2Compare)
Dim File2Reader 'As New StreamReader(OriginalFile)
Dim File3Writer As New StreamWriter(OutputFile)
Dim Line1 As String = ""
Dim Line2 As String = ""
Dim Found As Boolean

Do
Line1 = File1Reader.ReadLine
Found = False

If Not Line1 Is Nothing Then

File2Reader = New StreamReader(OriginalFile)

Do
Line2 = File2Reader.ReadLine()
If Line1 = Line2 Then
Found = True
Exit Do
End If
Loop Until Line2 Is Nothing

If Found = False Then
File3Writer.WriteLine(Line1)
End If

Found = False

File2Reader.Close()

End If
Loop Until Line1 Is Nothing

File1Reader.Close()
File2Reader.Close()
File3Writer.Close()

 

[Bob Hollness]

OK. This is the solution I came up with. Not as elegant as one would
have hoped. but then again, only I get to see how it functions under the
bonnet (hood for the Americans) !!! And of course, this is still to be
tidied up and made pretty. Feel free to pull it apart and embarrass
me.......


Sub FindDupes(ByVal File2Compare As String, ByVal OriginalFile As
String, ByVal OutputFile As String)

Dim File1Reader As New StreamReader(File2Compare)
Dim File2Reader 'As New StreamReader(OriginalFile)
Dim File3Writer As New StreamWriter(OutputFile)
Dim Line1 As String = ""
Dim Line2 As String = ""
Dim Found As Boolean

Do
Line1 = File1Reader.ReadLine
Found = False

If Not Line1 Is Nothing Then

File2Reader = New StreamReader(OriginalFile)

Do
Line2 = File2Reader.ReadLine()
If Line1 = Line2 Then
Found = True
Exit Do
End If
Loop Until Line2 Is Nothing

If Found = False Then
File3Writer.WriteLine(Line1)
End If

Found = False

File2Reader.Close()

End If
Loop Until Line1 Is Nothing

File1Reader.Close()
File2Reader.Close()
File3Writer.Close()

P.S. Yes I know that half the code is missing. It was late when I posted
this. I will update it with the missing parts this weekend.

 

[Richard Myers]

Thanks Cor. This looks a little too complicated and using too many CPU
cycles. I guess i'll loop each line through the each line of the other
file

If your concerned about performance then you shouldn;t be catching
exceptions as a means of handling a duplicate.
Exceptions are for exceptional circumstances and are, relatively speaking,
incredibly expensive.

You have posted some code below that is just a dowdy old serial search?

Almost everyone who has answered your thread has suggested a hash table,
why are you not using that?
If you dont understand hash tables then ask another question, but it seems
pointless to ask for a better way
and then say you cant be bothered learning how to use the better way?

Really? It just looks like so much more code. (Of course, currently, I
have no idea what a hashtable is.....!)

A hash table is simply a list that has been segmented. In order to search
for "Hollness" using your current method
in the phone book you basically start at A and go thru every letter of the
alphabet until you hit H. A hash table has
an algorithm that essentially allows you to skip letters on your way to H.
So rather than having to move through 8 letters
, you may only have to move thru 3 A > E > F.

Its an order of magnitude faster * the number of lines in your file.

Richard

 

[Bob Hollness]

If your concerned about performance then you shouldn;t be catching
exceptions as a means of handling a duplicate.
Exceptions are for exceptional circumstances and are, relatively speaking,
incredibly expensive.

You have posted some code below that is just a dowdy old serial search?

Almost everyone who has answered your thread has suggested a hash table,
why are you not using that?
If you dont understand hash tables then ask another question, but it seems
pointless to ask for a better way
and then say you cant be bothered learning how to use the better way?

Really? It just looks like so much more code. (Of course, currently, I
have no idea what a hashtable is.....!)

A hash table is simply a list that has been segmented. In order to search
for "Hollness" using your current method
in the phone book you basically start at A and go thru every letter of the
alphabet until you hit H. A hash table has
an algorithm that essentially allows you to skip letters on your way to H.
So rather than having to move through 8 letters
, you may only have to move thru 3 A > E > F.

Its an order of magnitude faster * the number of lines in your file.

Richard

OK, you got me. I have not been thinking straight. I have since given it
further thought and the hash table is clearly the better way to go,
especially because of the file sizes i eventually will be using. So, i am
currently writing it all now (or just customising the samples placed
here......? ;-) )

thanks for the code samples Cor, as always, you have been helpful.