What's the fastest way to combine two or more files in Linux?

Question

I want to combine two or more files in Linux, so I am using the following command:

cat small_file LARGE_File LARGER_FILE > SUM_OF_FILES

However this runs very slow.

Does anyone know a Linux tool that combines the files in the fastest time?

Solution 1

You could try a variation on the dd command, such as:

dd if=small_file bs=4k of=SUM_OF_FILES

dd if=LARGE_FILE bs=4k of=SUM_OF_FILES oflag=append

dd if=LARGER_FILE bs=4k of=SUM_OF_FILES oflag=append

Solution 2

I've found mmv (Mass Move and rename - Move, copy, append or link Multiple files using wildcard patterns.) from this useful bash reference. So you could do:

cp small_file SUM_OF_FILES
mmv -a LARGE_File SUM_OF_FILES
mmv -a LARGER_FILE SUM_OF_FILES

(note: mmv isn't installed by default, use sudo apt-get install mmv)

Comments

[vvj]

Get yourself faster hard drives. The bottleneck is not in the command, but in the speed of reading the data.

[Ed Heal]

How often do you need to do this? If frequently then consider a different system of running things.

[Admin]

@Ed Heal: periodically

[Keith Thompson]

@macki: "periodically" could mean 10 times per second or once every 10 years.

[Admin]

this is what I'm trying right now, I'm having problem with the exact size

[macki]

this is very slow when merging LARGE FILES

[suzanshakya]

brew install mmv to install mmv on max osx.