How do I use the lines of a file as arguments of a command?

Question

Say, I have a file foo.txt specifying N arguments

arg1
arg2
...
argN

which I need to pass to the command my_command

How do I use the lines of a file as arguments of a command?

Solution 1

If your shell is bash (amongst others), a shortcut for $(cat afile) is $(< afile), so you'd write:

mycommand "$(< file.txt)"

Documented in the bash man page in the 'Command Substitution' section.

Alterately, have your command read from stdin, so: mycommand < file.txt

Solution 2

As already mentioned, you can use the backticks or $(cat filename).

What was not mentioned, and I think is important to note, is that you must remember that the shell will break apart the contents of that file according to whitespace, giving each "word" it finds to your command as an argument. And while you may be able to enclose a command-line argument in quotes so that it can contain whitespace, escape sequences, etc., reading from the file will not do the same thing. For example, if your file contains:

a "b c" d

the arguments you will get are:

a
"b
c"
d

If you want to pull each line as an argument, use the while/read/do construct:

while read i ; do command_name $i ; done < filename

Solution 3

command `< file`

will pass file contents to the command on stdin, but will strip newlines, meaning you couldn't iterate over each line individually. For that you could write a script with a 'for' loop:

for line in `cat input_file`; do some_command "$line"; done

Or (the multi-line variant):

for line in `cat input_file`
do
    some_command "$line"
done

Or (multi-line variant with $() instead of ``):

for line in $(cat input_file)
do
    some_command "$line"
done

References:

1. For loop syntax: https://www.cyberciti.biz/faq/bash-for-loop/

Solution 4

You do that using backticks:

echo World > file.txt
echo Hello `cat file.txt`

Solution 5

If you want to do this in a robust way that works for every possible command line argument (values with spaces, values with newlines, values with literal quote characters, non-printable values, values with glob characters, etc), it gets a bit more interesting.

---

To write to a file, given an array of arguments:

printf '%s\0' "${arguments[@]}" >file

...replace with "argument one", "argument two", etc. as appropriate.

---

To read from that file and use its contents (in bash, ksh93, or another recent shell with arrays):

declare -a args=()
while IFS='' read -r -d '' item; do
  args+=( "$item" )
done <file
run_your_command "${args[@]}"

---

To read from that file and use its contents (in a shell without arrays; note that this will overwrite your local command-line argument list, and is thus best done inside of a function, such that you're overwriting the function's arguments and not the global list):

set --
while IFS='' read -r -d '' item; do
  set -- "$@" "$item"
done <file
run_your_command "$@"

Note that -d (allowing a different end-of-line delimiter to be used) is a non-POSIX extension, and a shell without arrays may also not support it. Should that be the case, you may need to use a non-shell language to transform the NUL-delimited content into an eval-safe form:

quoted_list() {
  ## Works with either Python 2.x or 3.x
  python -c '
import sys, pipes, shlex
quote = pipes.quote if hasattr(pipes, "quote") else shlex.quote
print(" ".join([quote(s) for s in sys.stdin.read().split("\0")][:-1]))
  '
}

eval "set -- $(quoted_list <file)"
run_your_command "$@"

View more solutions

Comments

[Charles Duffy]

"arguments", plural, or "argument", single? The accepted answer is correct only in the single-argument case -- which is what the body text inquires about -- but not in the multi-argument case, on which the topic appears to inquire.

[Navin]

The 4 answers below usually have identical results, yet they have slightly different semantics. Now I remember why I stopped writing bash scripts :P

[Will]

I should have mentioned, I am assuming that you are using bash. I realize that there are other shells out there, but almost all of the *nix machines I have worked on either ran bash or some equivalent. IIRC, this syntax should work the same on ksh and zsh.

[Charles Duffy]

Read should be read -r unless you want to expand backslash-escape sequences -- and NUL is a safer delimiter to use than the newline, particularly if the arguments you're passing are things like filenames, which can contain literal newlines. Also, without clearing IFS, you get leading and trailing whitespace implicitly cleared from i.

[Charles Duffy]

Your "nice little trick" is dangerous from a security perspective -- if you have an argument containing $(somecommand), you'll get that command executed rather than passed through as text. Likewise, >/etc/passwd will be processed as a redirection and overwrite /etc/passwd (if run with appropriate permissions), etc.

[Charles Duffy]

It's much safer to do the following instead (on a system with GNU extensions): xargs -d $'\n' sh -c 'for arg; do command1 "$arg"; command2 "arg"; command3 "arg"; done' _ -- and also more efficient, since it passes as many arguments to each shell as possible rather than starting one shell per line in your input file.

[Charles Duffy]

Also, putting < file in backticks means it doesn't actually perform a redirection for command at all.

[Charles Duffy]

This doesn't create an argument -- because it isn't quoted, it's subject to string-splitting, so if you emitted echo "Hello * Starry * World" > file.txt in the first step, you'd get at least four separate arguments passed to the second command -- and likely more, as the *s would expand to the names of files present in the current directory.

[Charles Duffy]

(Did the people who upvoted this actually test it? command `< file` doesn't work in either bash or POSIX sh; zsh is a different matter, but that's not the shell this question is about).

[Charles Duffy]

...and because it's running glob expansion, it doesn't emit exactly what's in the file. And because it's performing a command substitution operation, it's extremely inefficient -- it's actually fork()ing off a subshell with a FIFO attached to its stdout, then invoking /bin/cat as a child of that subshell, then reading the output through the FIFO; compare to $(<file.txt), which reads the file's contents into bash directly with no subshells or FIFOs involved.

[Charles Duffy]

This is -- but for being substantially less efficient -- effectively equivalent to $(<bar.txt) (when using a shell, such as bash, with the appropriate extension). $(<foo) is a special case: unlike regular command substitution, as used in $(cat ...), it doesn't fork off a subshell to operate in, and thus avoids a great deal of overhead.

[lstyls]

To be pedantic, it's not a shortcut to use the < operator. It means that the shell itself will perform the redirection rather than executing the cat binary for it's redirection properties.

[becko]

Is there a limit to the number of arguments? If so, how can I overcome it?

[glenn jackman]

It's a kernel setting so you'd need to recompile the kernel. Or investigate the xargs command

[mwfearnley]

@CharlesDuffy Can you be more specific about how it doesn't work?

[ykaner]

Can you explain to me why I need the $ here?

[glenn jackman]

Because it is required by the bash syntax as documented in the manual: Command Substitution

[törzsmókus]

@ykaner because without "$(…)", the contents of file.txt would be passed to the standard input of mycommand, not as an argument. "$(…)" means run the command and give back the output as a string; here the “command” only reads the file but it could be more complex.

[Pete]

It's not working for me unless I lose the quotes. With the quotes, it takes the whole file as 1 argument. Without quotes, it interprets each line as a separate arg.

[glenn jackman]

Without the quotes, it takes each word in the file as a separate arg, plus each word is subject to globbing - if one of the words is * then that expands to all the filenames in the current directory.

[tripleee]

And of course, lose the useless use of cat

[Arnie97]

@CharlesDuffy This does work in bash 3.2 and zsh 5.3. This won't work in sh, ash and dash, but neither $(< file).

[Charles Duffy]

@mwfearnley, happy to provide that specificity. The goal is to iterate over lines, right? Put Hello World on one line. You'll see it first run some_command Hello, then some_command World, not some_command "Hello World" or some_command Hello World.

[LarsH]

@lstyls: I'm not sure what you mean by "it's not a shortcut." Having the shell perform the redirection instead of spawning a child cat process sounds like it would save resources. The ref man describes this substitution as "equivalent but faster." Maybe you mean "it's not just a shorter way of typing the same thing"?

[lstyls]

@LarsH you’re absolutely correct. That’s a less confusing way to put it, thanks.

[glenn jackman]

So perhaps "a builtin alternative for" would be more accurate.

[Gabriel Staples]

The for loop option here works great! I just added some multi-line variants of it to help the next person. Tested it with calling git add on a bunch of lines to add that were stored in a file.

[Gabriel Staples]

Actually...it doesn't properly parse lines which have spaces in them. So, instead of editing this answer yet again and completely corrupting it from the author's original answer, I have added my own answer here! I like it the best of all now. It's completely borrowed from another site (and cited) though anyway. Here's my answer: stackoverflow.com/questions/4227994/….