removing duplicate lines from file with grep

Question

I want to remove all lines where the second column is 05408736032.

0009300|05408736032|89|01|001|0|0|0|1|NNNNNNYNNNNNNNNN|asdf|
0009367|05408736032|89|01|001|0|0|0|1|NNNNNNYNNNNNNNNN|adff|

Solution 1

This might do what you want:

sort -t '|' -k 2,2 -u  foo.dat

However this sorts the input according to your field, which you may not want. If you really only want to remove duplicates, your best option is Perl:

perl -ne '$a=(split "\\|")[1]; next if $h{$a}++; print;' foo.dat

Solution 2

awk -F \| '{if ($2 != 05408736032) print}'

Solution 3

Pure Bash:

oldIFS=$IFS
while read line
do
    IFS=$'|'
    testline=($line)  # make an array split according to $IFS
    IFS=$oldIFS       # put it back as soon as you can or you'll be sooOOoorry
    if [[ ${testline[1]} != "05408736032" ]]
    then
        echo $line
    fi
done < datafile

Comments

[Dennis Williamson]

Double posted: stackoverflow.com/questions/1439816/…

[Dennis Williamson]

You can leave out the "if" and the "print": awk -F \| '$2 != "05408736032"'