Remove Duplicate Lines in a Text File?

[No I'm Spartacus]

Hi,

Does anyone know of a utility to remove duplicate lines within a text
file? I've tried SlimList (*shudder*), and I looked at DupliFind -
which was recommended in this group a few years ago - but it seems to
be shareware. I don't want the program to sort the file, or change the
order of the entries. I want it to (preferably) highlight the
duplicate lines and let me decide on a line by line basis if I want to
delete them. I looked at ExamDiff too - but it looks like it will only
compare two files, not a single file.

 

[MaiBenRai]

Hi,

Does anyone know of a utility to remove duplicate lines within a text
file? I've tried SlimList (*shudder*), and I looked at DupliFind -
which was recommended in this group a few years ago - but it seems to
be shareware. I don't want the program to sort the file, or change the
order of the entries. I want it to (preferably) highlight the
duplicate lines and let me decide on a line by line basis if I want to
delete them. I looked at ExamDiff too - but it looks like it will only
compare two files, not a single file.
--

Regards,

Spartacus

Hi,

You could try uniq from TextUtils for Windows
http://gnuwin32.sourceforge.net/packages/textutils.htm

MaiBenRai

 

[Bill Day]

Hi,

Does anyone know of a utility to remove duplicate lines within a text
file? I've tried SlimList (*shudder*), and I looked at DupliFind -
which was recommended in this group a few years ago - but it seems to
be shareware. I don't want the program to sort the file, or change the
order of the entries. I want it to (preferably) highlight the
duplicate lines and let me decide on a line by line basis if I want to
delete them. I looked at ExamDiff too - but it looks like it will only
compare two files, not a single file.

get Columnizer
http://www.all4you.dk/FreewareWorld/Columnizer-2602.html

it has a menu item to "mark duplicate lines"...then you delete them.

 

[Bill Day]

get Columnizer
http://www.all4you.dk/FreewareWorld/Columnizer-2602.html

it has a menu item to "mark duplicate lines"...then you delete them.

(note...the 'markers' for duplicate lines are just numbers and
arrows... 1-->, 2-->...etc....and you must 'mouse over' the icon items
in the menu to find the right editing item. I would have made it a bit
easier to use, but then, it is the only app I have that will even FIND
duplicate lines.)

 

[No I'm Spartacus]

(note...the 'markers' for duplicate lines are just numbers and
arrows... 1-->, 2-->...etc....and you must 'mouse over' the icon items
in the menu to find the right editing item. I would have made it a bit
easier to use, but then, it is the only app I have that will even FIND
duplicate lines.)

Thanks Bill and MaiBenRai for the replies. I will give both those
programs a go. I will also try out CSVed - it apparently has a
duplicates option (my file is not a csv file, but it might still
work).