Command substitution: splitting on newline but not space
Solution 1
Looks like the canonical way to do this in bash
is something like
unset args
while IFS= read -r line; do
args+=("$line")
done < file
cmd "${args[@]}"
or, if your version of bash has mapfile
:
mapfile -t args < filename
cmd "${args[@]}"
The only difference I can find between the mapfile and the while-read loop versus the one-liner
(set -f; IFS=$'\n'; cmd $(<file))
is that the former will convert a blank line to an empty argument, while the one-liner will ignore a blank line. In this case the one-liner behavior is what I'd prefer anyway, so double bonus on it being compact.
I would use IFS=$'\n' cmd $(<file)
but it doesn't work, because $(<file)
is interpreted to form the command line before IFS=$'\n'
takes effect.
Though it doesn't work in my case, I've now learned that a lot of tools support terminating lines with null (\000)
instead of newline (\n)
which does make a lot of this easier when dealing with, say, file names, which are common sources of these situations:
find / -name '*.config' -print0 | xargs -0 md5
feeds a list of fully-qualified file names as arguments to md5 without any globbing or interpolating or whatever. That leads to the non-built-in solution
tr "\n" "\000" <file | xargs -0 cmd
although this, too, ignores empty lines, though it does capture lines that have only whitespace.
Solution 2
Portably:
set -f # turn off globbing
IFS='
' # split at newlines only
cmd $(cat <file)
unset IFS
set +f
Or using a subshell to make the IFS
and option changes local:
( set -f; IFS='
'; exec cmd $(cat <file) )
The shell performs field splitting and filename generation on the result of a variable or command substitution that is not in double quotes. So you need to turn off filename generation with set -f
, and configure field splitting with IFS
to make only newlines separate fields.
There's not much to be gained with bash or ksh constructs. You can make IFS
local to a function, but not set -f
.
In bash or ksh93, you can store the fields in an array, if you need to pass them to multiple commands. You need to control expansion at the time you build the array. Then "${a[@]}"
expands to the elements of the array, one per word.
set -f; IFS=$'\n'
a=($(cat <file))
set +f; unset IFS
cmd "${a[@]}"
Solution 3
You can do this with a temporary array.
Setup:
$ cat input
AAA
A B C
DE F
$ cat t.sh
#! /bin/bash
echo "$1"
echo "$2"
echo "$3"
Fill the array:
$ IFS=$'\n'; set -f; foo=($(<input))
Use the array:
$ for a in "${foo[@]}" ; do echo "--" "$a" "--" ; done
-- AAA --
-- A B C --
-- DE F --
$ ./t.sh "${foo[@]}"
AAA
A B C
DE F
Can't figure out a way of doing that without that temporary variable - unless the IFS
change isn't important for cmd
, in which case:
$ IFS=$'\n'; set -f; cmd $(<input)
should do it.
Solution 4
You could use the bash built-in mapfile
to read the file into an array
mapfile -t foo < filename
cmd "${foo[@]}"
or, untested, xargs
might do it
xargs cmd < filename
Related videos on Youtube
Old Pro
Updated on September 18, 2022Comments
-
Old Pro almost 2 years
I know I can solve this problem several ways, but I'm wondering if there is a way to do it using only bash built-ins, and if not, what is the most efficient way to do it.
I have a file with contents like
AAA B C DDD FOO BAR
by which I only mean it has several lines and each line may or may not have spaces. I want to run a command like
cmd AAA "B C DDD" "FOO BAR"
If I use
cmd $(< file)
I getcmd AAA B C DDD FOO BAR
and if I use
cmd "$(< file)"
I getcmd "AAA B C DDD FOO BAR"
How do I get each line treated a exactly one parameter?
-
Old Pro about 12 yearsFrom the mapfile documentation: "mapfile isn't a common or portable shell feature". And indeed is it not supported on my system.
xargs
doesn't help, either. -
Batfan about 12 yearsYou would need
xargs -d
orxargs -L
-
Old Pro about 12 years@James, no, I don't have a
-d
option andxargs -L 1
runs the command once per line but still splits args on whitespace. -
Gilles 'SO- stop being evil' about 12 years@OldPro
IFS=$'\n' cmd $(<input)
doesn't work because it only setsIFS
in the environment ofcmd
.$(<input)
is expanded to form the command, before the assignment toIFS
is performed. -
Angel Todorov about 12 years@OldPro, well you did ask for "a way to do it using only bash built-ins" instead of "a common or portable shell feature". If your version of bash is too old, can you update it?
-
Peter.O about 12 years
mapfile
is very handy for me, as it grabs blank lines as array items, which theIFS
method does not do.IFS
treats contiguous newlines as a single delimiter... Thanks for presenting it, as I wasn't aware of the command (though, based on the OP's input data and the expected command line, it seems he actually wants to ignore blank lines). -
Batfan about 12 yearsI introduced the
-d
option in 2005, your version ofxargs
must be quite old. You're right about-L
; I should have typed-I{}
(but even that does unexpected things with trailing spaces, which is one of the reasons-d
exists). -
Dirk Thannhäuser almost 9 yearsUsing
cmd $(<file)
values without quoting (using the ability of bash to split words) is always a risky bet. If any line is*
it will be expanded by the shell to a list of files. -
RalfFriedl over 5 yearsAnd why does it work if you set
IFS
to space, but the question is to not split on space?