Quoting within $(command substitution) in Bash

Question

In my Bash environment I use variables containing spaces, and I use these variables within command substitution.

What is the correct way to quote my variables? And how should I do it if these are nested?

DIRNAME=$(dirname "$FILE")

or do I quote outside the substitution?

DIRNAME="$(dirname $FILE)"

or both?

DIRNAME="$(dirname "$FILE")"

or do I use back-ticks?

DIRNAME=`dirname "$FILE"`

What is the right way to do this? And how can I easily check if the quotes are set right?

Solution 1

In order from worst to best:

• DIRNAME="$(dirname $FILE)" will not do what you want if $FILE contains whitespace (or whatever characters $IFS currently contains) or globbing characters \[?*.
• DIRNAME=`dirname "$FILE"` is technically correct, but backticks are not recommended for command expansion because of the extra complexity when nesting them and the extra backslash processing that happens within them.
• DIRNAME=$(dirname "$FILE") is correct, but only because this is an assignment to a scalar (not array) variable. If you use the command substitution in any other context, such as export DIRNAME=$(dirname "$FILE") or du $(dirname -- "$FILE"), the lack of quotes will cause trouble if the result of the expansion contain whitespace or globbing characters.
• DIRNAME="$(dirname "$FILE")" (except for the missing --, see below) is the recommended way. You can replace DIRNAME= with a command and a space without changing anything else, and dirname receives the correct string.

To improve even further:

• DIRNAME="$(dirname -- "$FILE")" works if $FILE starts with a dash.
• DIRNAME="$(dirname -- "$FILE" && printf x)" && DIRNAME="${DIRNAME%?x}" || exit works even if $FILE's dirname ends with a newline, since $() chops off newlines at the end of output, both the one added by dirname and the ones that may be part of the actual data.

You can nest command expansions as much as you like. With $() you always create a new quoting context, so you can do things like this:

foo "$(bar "$(baz "$(ban "bla")")")"

You do not want to try that with backticks.

Solution 2

You can always show the effects of variable quoting with printf.

Word splitting done on var1:

$ var1="hello     world"
$ printf '[%s]\n' $var1
[hello]
[world]

var1 quoted, so no word splitting:

$ printf '[%s]\n' "$var1"
[hello     world]

Word splitting on var1 inside $(), equivalent to echo "hello" "world":

$ var2=$(echo $var1)
$ printf '[%s]\n' "$var2"
[hello world]

No word splitting on var1, no problem with not quoting the $():

$ var2=$(echo "$var1")
$ printf '[%s]\n' "$var2"
[hello     world]

Word splitting on var1 again:

$ var2="$(echo $var1)"
$ printf '[%s]\n' "$var2"
[hello world]

Quoting both, easiest way to be sure.

$ var2="$(echo "$var1")"
$ printf '[%s]\n' "$var2"
[hello     world]

Globbing problem

Not quoting a variable can also lead to glob expansion of its contents:

$ mkdir test; cd test; touch file1 file2
$ var="*"
$ printf '[%s]\n' $var
[file1]
[file2]
$ printf '[%s]\n' "$var"
[*]

Note this happens after the variable is expanded only. It is not necessary to quote a glob during assignment:

$ var=*
$ printf '[%s]\n' $var
[file1]
[file2]
$ printf '[%s]\n' "$var"
[*]

Use set -f to disable this behaviour:

$ set -f
$ var=*
$ printf '[%s]\n' $var
[*]

And set +f to re-enable it:

$ set +f
$ printf '[%s]\n' $var
[file1]
[file2]

Solution 3

Addition to the accepted answer:

While I generally agree with @l0b0's answer here, I suspect the placement of bare backticks in the "worst to best" list is at least partly a result of the assumption that $(...) is available everywhere. I realize that the question specifies Bash, but there are plenty of times when Bash turns out to mean /bin/sh, which may not always actually be the full Bourne Again shell.

In particular, the plain Bourne shell won't know what to do with $(...), so scripts which claim to be compatible with it (e.g., via a #!/bin/sh shebang line) will likely misbehave if they are actually run by the "real" /bin/sh – this is of special interest when, say, producing init scripts, or packaging pre- and post-scripts, and can land one in a surprising place during installation of a base system.

If any of that sounds like something you're planning to do with this variable, nesting is probably less of a concern than having the script actually, predictably run. When it's a simple enough case and portability is a concern, even if I expect the script to usually run on systems where /bin/sh is Bash, I often tend to use backticks for this reason, with multiple assignments instead of nesting.

Having said all that, the ubiquity of shells which implement $(...) (Bash, Dash, et al.), leaves us in a good spot to stick with the prettier, easier-to-nest, and more recently preferred POSIX syntax in most cases, for all the reasons @l0b0 mentions.

Aside: this has shown up occasionally on StackOverflow, too –

• Command substitution: backticks or dollar sign / paren enclosed? [duplicate] (Feb 2012)
• Shell Programming: What's the difference between $(command) and `command` (Jan 2011)

Comments

[Graeme]

See also - unix.stackexchange.com/questions/97560/…

[Gilles 'SO- stop being evil']

See also When is double-quoting necessary? and $VAR vs ${VAR} and to quote or not to quote

[gokhan acar]

This is a good question, but given all the issues with embedded blanks, why would you make life hard on yourself by using them on purpose?

[Conner Dassen]

@Joe, with embedded blanks you mean space in the filenames? Personally I do not use them that often, but I am working with other peoples directories and files of which I am not certain if they contain spaces. Furthermore, I think it is better to get it right at once so I do not have to worry in the future.

[gokhan acar]

Yes. What are we going to do with those "other people"? <G>

[Stéphane Chazelas]

People tend to forget that word splitting is not the only problem, you may want to change your example to have var1='hello * world' to illustrate the globbing problem as well.

[AmadeusDrZaius]

Is there a reference/resource detailing the behavior of quotes within a command substition within quotes?

[l0b0]

@AmadeusDrZaius "With $() you always create a new quoting context", so it's just like outside the outer quotes. There's nothing more to it, as far as I know.

[AmadeusDrZaius]

@l0b0 Thanks, yeah, I found your explanation very clear. I was just wondering whether it was in a manual somewhere as well. I did find it (albeit unofficially) at wooledge. I guess if you read about order of substitution carefully, you could derive this fact as a result.

[Nathan Basanese]

// , Excellent answer. I have run into backwards compatibility issues with /bin/sh in the past, too. Do you have any advice about how to deal with his problem using backwards compatible methods?

[Luke Davis]

So nested quotes are acceptable, but they throw us off because most syntax coloring schemes don't detect the special circumstance. Neat.

[dma_k]

Is it safe to have the deep-most execution as backticks? E.g. foo "$(bar "$(baz "`ban "bla"`")")"

[l0b0]

@dma_k Sure, but why would you? It's inconsistent, more complicated, and makes for a bigger diff if you ever change the nesting.

[dma_k]

I never used more than two "inclusions". And the place where it hurts is Makefile where dollar sign needs to be escaped $$ (error-prone when it comes to copying from bash scripts), that is why I prefer backticks which is more "compatible" in the sense.

[g4v3]

+1 for the quoted, nested command substitutions example at the end.

[Steven Lu]

in my experience: sometimes you might need to change backticks into dollar paren enclosed, and not once has it ever helped (been necessary, to be specific) to change dollar paren enclosed into backticks. I only write backticks in my javascript code, and not in shell code.