Concatenate multiple files but include filename as section headers

Question

I would like to concatenate a number of text files into one large file in terminal. I know I can do this using the cat command. However, I would like the filename of each file to precede the "data dump" for that file. Anyone know how to do this?

what I currently have:

file1.txt = bluemoongoodbeer

file2.txt = awesomepossum

file3.txt = hownowbrowncow

cat file1.txt file2.txt file3.txt

desired output:

file1

bluemoongoodbeer

file2

awesomepossum

file3

hownowbrowncow

Solution 1

Was looking for the same thing, and found this to suggest:

tail -n +1 file1.txt file2.txt file3.txt

Output:

==> file1.txt <==
<contents of file1.txt>

==> file2.txt <==
<contents of file2.txt>

==> file3.txt <==
<contents of file3.txt>

If there is only a single file then the header will not be printed. If using GNU utils, you can use -v to always print a header.

Solution 2

I used grep for something similar:

grep "" *.txt

It does not give you a 'header', but prefixes every line with the filename.

Solution 3

This should do the trick as well:

$ find . -type f -print -exec cat {} \;
./file1.txt
Content of file1.txt
./file2.txt
Content of file2.txt

Here is the explanation for the command-line arguments:

find    = linux `find` command finds filenames, see `man find` for more info
.       = in current directory
-type f = only files, not directories
-print  = show found file
-exec   = additionally execute another linux command
cat     = linux `cat` command, see `man cat`, displays file contents
{}      = placeholder for the currently found filename
\;      = tell `find` command that it ends now here

You further can combine searches trough boolean operators like -and or -or. find -ls is nice, too.

Solution 4

This should do the trick:

for filename in file1.txt file2.txt file3.txt; do
    echo "$filename"
    cat "$filename"
done > output.txt

or to do this for all text files recursively:

find . -type f -name '*.txt' -print | while read filename; do
    echo "$filename"
    cat "$filename"
done > output.txt

Solution 5

When there is more than one input file, the more command concatenates them and also includes each filename as a header.

To concatenate to a file:

more *.txt > out.txt

To concatenate to the terminal:

more *.txt | cat

Example output:

::::::::::::::
file1.txt
::::::::::::::
This is
my first file.
::::::::::::::
file2.txt
::::::::::::::
And this is my
second file.

View more solutions

Comments

[Nick]

didn't work. I just wrote some really ugly awk code: for i in $listoffiles do awk '{print FILENAME,$0,$1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11}' $i >> concat.txt done

[Chris Eberle]

...care to elaborate? That's about as simple as bash code gets.

[Chris Eberle]

@Nick: your awk line shouldn't even work, considering that $0 is the entire line, so you've actually got repeating columns in there...

[Chris Eberle]

@Nick: Nifty solution otherwise :)

[Nick]

@Chris: yes, but its a lot uglier than I would like it to be. Maybe your code wasn't working for me because I'm using >> to catch the stdout?

[Chris Eberle]

@Nick: the other possibility is that your list of filenames have spaces. for in bash is rather dumb and splits on any whitespace.

[jayunit100]

make sure not to use a * as the wildcard... it winds up infinitely recurring for some reason.

[ArjunShankar]

This works with the GNU tail (part of GNU Coreutils) as well.

[doubleDown]

a cleaner syntax would be head -n -0 file.txt file2.txt file3.txt. Works for head in GNU coreutils

[Alec Jacobson]

I want to concatenate a large number of files. If I use this (tail +1 *.txt) I get an error Too many open files. I fixed this with: OLD_ULIMIT=`ulimit -n`;ulimit -n 1024;tail +1 *.txt;ulimit -n $OLD_ULIMIT;. Where 1024 was large enough for me.

[Asclepius]

Be careful here. for i in * won't include subdirectories.

[AAlvz]

Could you explain more what this command does? Is exactly what I Needed

[AAlvz]

and actually it works fine also without the -print instruction

[rshetye]

@jayunit100: the infinite recurring is if your target file matches the wild card. name it such that it does not match.

[Maxim_united]

This is linux' standard find command. It searches all files in the current directory, prints their name, then for each one, cats the file. Omitting the -print won't print the filename before the cat.

[Maxim_united]

You can also use -printf to customize the output. For example: find *.conf -type f -printf '\n==> %p <==\n' -exec cat {} \; to match the output of tail -n +1 *

[antak]

Output breaks if *.txt expands to only one file. In this regard, I'd advise grep '' /dev/null *.txt

[verboze]

+1 for showing me a new use for grep. this met my needs perfectly. in my case, each file only contained one line, so it gave me a neatly formatted output that was easily parsable

[Matt Fletcher]

I like this method as it gives much more flexibility with how you want to present the output!

[dojuba]

It is also quite straightforward to combine with grep. To use with bash: find . -type f -print | grep PATTERN | xargs -n 1 -I {} -i bash -c 'echo ==== {} ====; cat {}; echo'

[tripleee]

The -f is useful if you want to track a file which is being written to, but not really within the scope of what the OP asked.

[Jorge Bucaran]

grep will only print file headers if there is more than one file. If you want to make sure to print the file path always, use -H. If you don't want the headers use -h. Also note it will print (standard input) for STDIN.

[Frozen Flame]

Awesome -n +1 option! An alternative: head -n-0 file1 file2 file3.

[Matt Fletcher]

-printf doesn't work on mac, unless you want to brew install findutils and then use gfind instead of find.

[Fekete Sumér]

Explanation: for loop goes through the list the ls command resulted. $ ls file{1..3}.txt Result: file1.txt file2.txt file3.txt each iteration will echo the $file string then it's piped into a cut command where I used . as a field separator which breaks fileX.txt into two pieces and prints out the field1 (field2 is the txt) The rest should be clear

[anol]

You can also use ag (the silver searcher): by default, ag . *.txt prefixes each file with its name, and each line with its number.

[MediaVince]

For the sake of colors highlighting the filename: find . -type f -name "*.txt" | xargs -I {} bash -c "echo $'\e[33;1m'{}$'\e[0m';cat {}"

[Basic]

If you want to reference a different directory, you can control the name relative root with this one liner... (cd /var/log; grep "" */*.log). This will print names relative to /var/log

[DepressedDaniel]

Of note, passing -n to grep also yields line numbers which allows you to write simple linters with pinpointing that could be picked up by, e.g., emacs.

[vdm]

Works great with both BSD tail and GNU tail on MacOS X. You can leave out the space between -n and +1, as in -n+1.

[kR105]

tail -n +1 * was exactly was I was looking for, thanks!

[Patrick M]

cat from GNU coreutils 8.22 does not include the header by default, and does not appear to have an option for doing so. Combining your find syntax with DS.'s answer, the command find . -type f -name file* -exec tail -n +1 {} +; works exactly.

[Bar]

That, or ls | xargs tail -n +1 should also work fine.

[JasonGenX]

nice one. I wasn't aware of it.

[banbh]

If you want colors you can use that fact that find allows multiple -execs: find -name '*.conf' -exec printf '\n\e[33;1m%s\e[0m\n' {} \; -exec cat {} \;

[kolas]

works on MacOsX 10.14.4 sudo ulimit -n 1024; find -f . -name "*.rb" | xargs tail -n+1 > ./source_ruby.txt

[kvantour]

But this does not answer the OP's question.

[bwangel]

Thanks for your great answer. Can you explain the meaning of the 1 at the end of expression?

[kvantour]

Thanks. You can have a look at what is the meaning of 1 at the end of awk script

[Coddy]

Also if you need to concatenate without file names use -q (for silent) flag with head or tail command.

[mblakesley]

I've always found it bizarre that this is the default behavior for head & tail, yet it's not even an option for cat. AND it still hasn't been added after all these years. Like, what?? I use this so often that I added alias cats='head -n -0'

[WinEunuuchs2Unix]

@Acumenus For myself I had to use: String="${String//$'\n'::::::::::::::$'\n'/|}" then: String="${String//::::::::::::::$'\n'/}" and finally: String="${String//$'\n'/|}" to make into a YAD array: IFS='|' read -ra OLD_ARR <<< "$String"

[WinEunuuchs2Unix]

@Acumenus First I had to build the string field using: String=$(sudo -u "$SUDO_USER" ssh "$SUDO_USER"@"$TRG_HOST" \ "find /tmp/$SUDO_USER/scp.*/*.Header -type f \ -printf '%Ts\t%p\n' | sort -nr | cut -f2 | \ xargs more | cat | cut -d'|' -f2,3" \ )

[Asclepius]

@WinEunuuchs2Unix I don't follow. If you'd like, you can explain more clearly in a gist.github.com, and link to it instead.

[WinEunuuchs2Unix]

@Acumenus Better yet I'll upload the script to github when done. It's just to copy root files between hosts using sudo because hosts don't have root accounts. The code in comments was to select headers for previous payloads. Kind of a fringe thing that won't interest most users.

[mgutt]

I tried tail -n +1 * as well, but it does not skip dirs and will be interrupted by them. Is there a way to skip dirs?

[stason]

This solution only prints the last few lines - not the whole files' contents.

[ben-albrecht]

This does not work for me on MacOS. I only get the file contents.

[young_souvlaki]

As a bash function (SO comment doesn't support line breaks): tails() { tail -v -n +1 "$@" | less }.

[sourabh kesharwani]

Cool trick. I got the output as I needed. Thanks.