Merge two lists while removing duplicates
Solution 1
In just one command without any pipe :
sort -u FILE1 FILE2
search
Suppress duplicate lines
-> http://www.busybox.net/downloads/BusyBox.html
Solution 2
Another solution:
awk '!a[$0]++' file_1 file_2
Solution 3
To sort according to some key column use following :
awk '!duplicate[$1,$2,$3]++' file_1 file_2
here consider first, second and third column as your primary key.
Solution 4
The files on your question are sorted.
If the source files are indeed sorted, you can uniq and merge in one step:
sort -um file1 file2 > mylist.merge
For numeric sort (not alphanumeric), use:
sort -num file1 file2 > mylist.merge
That could not be done in-place (redirected to one source file).
If the files are not sorted, sort them (this sort could be done in place, using the sort option -o
. However, the whole file needs to be loaded into memory):
sort -uo file1 file1
sort -uo file2 file2
sort -um file1 file2 > mylist.merge
mv mylist.merge originallist
That would be faster than the simpler "one command line" to sort all:
cat file1 file2 | sort -u >mylist.merge
However, this line could be useful for small files.
Related videos on Youtube
learningbee
Updated on September 18, 2022Comments
-
learningbee over 1 year
I have an embedded linux system using Busybox (OpenWRT) - so commands are limited. I have two files that look like:
first file
aaaaaa bbbbbb cccccc mmmmmm nnnnnn
second file
mmmmmm nnnnnn yyyyyy zzzzzz
I need to merge these 2 lists into 1 file, and remove the duplicates. I don't have diff (space is limited) so we get to use the great
awk
,sed
, andgrep
(or other tools that might be included in a standard Busybox instance). Going to a merge file like:command1 > mylist.merge command2 mylist.merge > originallist
is totally ok. It doesn't have to be a single-line command.
Currently defined functions in the instance of Busybox that I am using (default OpenWRT): [, [[, arping, ash, awk, basename, brctl, bunzip2, bzcat, cat, chgrp, chmod, chown, chroot, clear, cmp, cp, crond, crontab, cut, date, dd, df, dirname, dmesg, du, echo, egrep, env, expr, false, fgrep, find, free, fsync, grep, gunzip, gzip, halt, head, hexdump, hostid, hwclock, id, ifconfig, init, insmod, kill, killall, klogd, less, ln, lock, logger, logread, ls, lsmod, md5sum, mkdir, mkfifo, mknod, mktemp, mount, mv, nc, netmsg, netstat, nice, nslookup, ntpd, passwd, pgrep, pidof, ping, ping6, pivot_root, pkill, poweroff, printf, ps, pwd, reboot, reset, rm, rmdir, rmmod, route, sed, seq, sh, sleep, sort, start-stop-daemon, strings, switch_root, sync, sysctl, syslogd, tail, tar, tee, telnet, telnetd, test, time, top, touch, tr, traceroute, true, udhcpc, umount, uname, uniq, uptime, vconfig, vi, watchdog, wc, wget, which, xargs, yes, zcat
-
Admin over 11 yearsbusybox sort supports the unique flag
-u
. -
Admin over 11 years@Thor: oooh cheers that's not a switch I'm familiar with.
-
dza over 7 yearsI saw that it made a difference which argument came first. Otherwise great solution, thanks.
-
Pat over 5 yearswhich one is better for very large files?
sort file1 file2 file3 file4 | uniq
orsort -u file1 file2 file3 file4
-
damd over 3 yearsIt boggles my mind that if someone posts e.g. a Python solution to a problem, people often request an explanation of the code, but whenever it's awk, which relatively few people understand, it's fine to just leave it as is. :)