Recursively cat all the files into single file

Question

I have bunch of files sitting in folders like

data\A\A\A\json1.json
data\A\A\A\json2.json
data\A\A\B\json1.json
...
data\Z\Z\Z\json_x.json

I want to cat all the jsons into one single file?

Solution 1

find data/ -name '*.json' -exec cat {} \; > uber.json

a short explanation:

find <where> \
  -name <file_name_pattern> \
  -exec <run_cmd_on_every_hit> {} \; \
    > <where_to_store>

Solution 2

Use find to get all the JSON files and concatenate them.

find data -name '*.json' -exec cat {} + > all.json

Note that this will not be valid JSON. If you want a JSON file to contain multiple objects, they need to be in a containing array or object, so you'd need to add [ ] around them and put , between each one.

Solution 3

Alternatively -- if you have a list of your files -- you can pipe that to xargs

<path to your files> | xargs cat > all.json

Solution 4

find ./ -type f | xargs cat > ../singlefilename

I would like this ,easy and simple.

../ avoid the error input file is output file.

View more solutions

Comments

[Mr. Llama]

Without using find: cat ./data/*/*/*/*.json > ./all.json

[chepner]

Or in bash 4 with shopt -s globstar, cat ./data/**/*.json.

[Barmar]

Strictly true, but it would be quite perverse to have directories with .json extensions.

[Alan]

to get a list recursively: find `pwd`

[Bojan Hrnkas]

You should also add "-type f"(only files) to avoid errors when cat tries to print a directory.

[Gal Bracha]

How do you add the ',' chars in the command above between each file?

[Barmar]

@GalBracha Use a find command that just prints the filenames, pipe that to a script that reads each filename, does cat "$filename"; echo ','.

[Barmar]

You'll then have to edit the file when done to remove the last ,

[rwitzel]

What is \; doing?

[Barmar]

@rwitzel If you end the -exec option with + instead of ; it replaces {} with all the filenames instead of running the command separately for each filename.

[ZaydH]

@rwitzel ; ends the exec command. You need to escape it with `` though otherwise it would end the entire command. More info: askubuntu.com/questions/339015/…

[coyot]

find ./ -type f | fgrep -v singlefilename | xargs cat > ../singlefilename