Why does shell ignore quoting characters in arguments passed to it through variables?

10,504

Solution 1

Why

When the string is expanded, it is split into words, but it is not re-evaluated to find special characters such as quotes or dollar signs or ... This is the way the shell has 'always' behaved, since the Bourne shell back in 1978 or thereabouts.

Fix

In bash, use an array to hold the arguments:

argumentArray=(-ir 'hello world')
grep "${argumentArray[@]}" .

Or, if brave/foolhardy, use eval:

argumentString="-ir 'hello world'"
eval "grep $argumentString ."

On the other hand, discretion is often the better part of valour, and working with eval is a place where discretion is better than bravery. If you are not completely in control of the string that is eval'd (if there's any user input in the command string that has not been rigorously validated), then you are opening yourself to potentially serious problems.

Note that the sequence of expansions for Bash is described in Shell Expansions in the GNU Bash manual. Note in particular sections 3.5.3 Shell Parameter Expansion, 3.5.7 Word Splitting, and 3.5.9 Quote Removal.

Solution 2

When you put quote characters into variables, they just become plain literals (see http://mywiki.wooledge.org/BashFAQ/050; thanks @tripleee for pointing out this link)

Instead, try using an array to pass your arguments:

argumentString=(-ir 'hello world')
grep "${argumentString[@]}" .

Solution 3

In looking at this and related questions, I'm surprised that no one brought up using an explicit subshell. For bash, and other modern shells, you can execute a command line explicitly. In bash, it requires the -c option.

argumentString="-ir 'hello world'"
bash -c "grep $argumentString ."

Works exactly as original questioner desired. There are two restrictions to this technique:

  1. You can only use single quotes within the command or argument strings.
  2. Only exported environment variables will be available to the command

Also, this technique handles redirection and piping, and other shellisms work as well. You also can use bash internal commands as well as any other command that works at the command line, because you are essentially asking a subshell bash to interpret it directly as a command line. Here's a more complex example, a somewhat gratuitously complex ls -l variant.

cmd="prefix=`pwd` && ls | xargs -n 1 echo \'In $prefix:\'"
bash -c "$cmd"

I have built command processors both this way and with parameter arrays. Generally, this way is much easier to write and debug, and it's trivial to echo the command you are executing. OTOH, param arrays work nicely when you really do have abstract arrays of parameters, as opposed to just wanting a simple command variant.

Share:
10,504
Ben Wilhelm
Author by

Ben Wilhelm

Updated on June 11, 2022

Comments

  • Ben Wilhelm
    Ben Wilhelm almost 2 years

    These work as advertised:

    grep -ir 'hello world' .
    grep -ir hello\ world .
    

    These don't:

    argumentString1="-ir 'hello world'"
    argumentString2="-ir hello\\ world"
    grep $argumentString1 .
    grep $argumentString2 .
    

    Despite 'hello world' being enclosed by quotes in the second example, grep interprets 'hello (and hello\) as one argument and world' (and world) as another, which means that, in this case, 'hello will be the search pattern and world' will be the search path.

    Again, this only happens when the arguments are expanded from the argumentString variables. grep properly interprets 'hello world' (and hello\ world) as a single argument in the first example.

    Can anyone explain why this is? Is there a proper way to expand a string variable that will preserve the syntax of each character such that it is correctly interpreted by shell commands?

    • nneonneo
      nneonneo over 11 years
      I should note this doesn't have anything to do with grep per se; it's more of a bash issue (using any other command has the same effect)
  • tripleee
    tripleee over 11 years
    Or, "don't do that then". mywiki.wooledge.org/BashFAQ/050
  • Charles Duffy
    Charles Duffy about 7 years
    If we're going to showcase using eval, might as well showcase using it right. I'd argue that using eval right always consists of passing it a single string -- otherwise, you're concatenating all its arguments together with whitespace, and that gets messy in surprising ways. Consider eval printf '%s\n' "hello world", compared to eval 'printf "%s\n" "hello world"', for a case-in-point example of why passing eval multiple arguments leads to confusion.
  • natevw
    natevw about 6 years
    Why don't the quotes in grep "${argumentArray[@]}" . cause the array to get passed as a single argument? I would have expected it to be grep ${argumentArray[@]} . instead, but it seems that either works in practice?
  • Jonathan Leffler
    Jonathan Leffler about 6 years
    @natevw: Roughly for the same reason that "$@" produces a list of arguments not a single string (and "$*" produces a single string, as does "${array[*]}"). Why does "$@" do that? Because it's been like that since time immemorial (or, at least, 7th Edition Unix, circa 1978, which introduced the Bourne shell).
  • that other guy
    that other guy over 5 years
    @JonathanLeffler I want a canonical FAQ for this answer that is not restricted to quotes. Should I post a new question instead?
  • Jonathan Leffler
    Jonathan Leffler over 5 years
    @thatotherguy: No; post a new answer that gives the answer you want given. If it's good, it will rise up the rankings over time. Please leave my answer as it is, though — I don't understand all the changes you've tried to make, so it isn't my answer if you make the changes.
  • that other guy
    that other guy over 5 years
    I edited the question to be more generally applicable as well. Should I revert that too? The intent here was to make the question and answer more applicable and helpful to more people, and make it easier to dedupe questions on this topic that are not strictly related to quotes, because they show up pretty frequently. Again I'm happy to repost a self-answered and more general version of this question if you prefer that.
  • jrw32982
    jrw32982 about 2 years
    Both single and double quotes can be used within the command. Just like when you're typing at the command line. cmd='x=" a "'\''$PATH'\''" b "'; echo "<$cmd>"; bash -c "$cmd; echo \"\$x\""