How can I run a "grep | grep" command as a string in a bash function?

Question

I'm trying to build a command that pipes the results of one grep command to another grep command in a bash function. Ultimately, I want the command executed to look like this:

grep -I -r FooBar /code/internal/dev/ /code/public/dev/ | grep .c:\|.h:

The function I'm writing stores the first part of the command in a string, then appends the second part:

grep_cmd="grep -I -r $pattern $@"

if (( ${#file_types[@]} > 0 )); then
    file_types="${file_types[@]}"
    file_types=.${file_types// /':\|.'}:

    grep_cmd="$grep_cmd | grep $file_types"
fi

echo "$grep_cmd"
${grep_cmd}

This throws an error after the output from the first part:

grep: |: No such file or directory
grep: grep: No such file or directory
grep: .c:\|.h:: No such file or directory

Changing the last line from ${grep_cmd} to just "$grep_cmd" displays no output from the first part and throws a different error:

bash: grep -I -r FooBar /code/internal/dev/ /code/public/dev/ | grep .c:\|.h:: No such file or directory

Per this SO answer, I tried changing the last line to $(grep_cmd). This throws another error:

bash: grep_cmd: command not found

This SO answer suggests using eval $grep_cmd. This suppresses the errors but also suppresses the output.

This one suggests using eval ${grep_cmd}. This has the same results (suppresses the errors and output). I tried enabling debugging in bash (with set -x), which gives me this:

+ eval grep -I -r FooBar /code/internal/dev/ /code/public/dev/ '|' grep '.c:\|.h:'
++ grep -I -r FooBar /code/internal/dev/ /code/public/dev/
++ grep '.c:|.h:'

It looks like the pipe is being escaped, so the shell interprets the command as two commands. How do I properly escape the pipe character so it interprets it as one command?

Solution 1

As mentioned in the comment, a lot of your difficulty is because you're trying to store a command in a variable, and then run that command later.

You'll have a lot better luck if you just run the command immediately instead of trying to save it.

For example, this should do what you're trying to accomplish:

if (( ${#file_types[@]} > 0 )); then
    regex="${file_types[*]}"
    regex="\.\(${regex// /\|}\):"
    grep -I -r "$pattern" "$@" | grep "$regex"
else
    grep -I -r "$pattern" "$@"
fi

Solution 2

One thing to keep in mind about shell programming, which is often not explained clearly in tutorials, is that there are two types of data: strings, and lists of strings. A list of strings is not the same thing as strings with a newline or space separator, it's a thing of its own.

Another thing to keep in mind is that most expansions are only applied when the shell is parsing a file. Executing a command does not involve any expansion.

Some expansion does happen to the value of a variable: $foo means “take the value of the variable foo, split it into a list of strings using whitespace¹ as separators, and interpret each element of the list as a wildcard pattern which is then expanded”. This expansion only happens if the variable is used in a context that calls for a list. In a context that calls for a string, $foo means “take the value of the variable foo”. Double quotes impose a string context, hence the advice: always use variable substitutions and command substitutions in double quotes: "$foo", "$(somecommand)"². (The same expansions happen to unprotected command substitutions as to variables.)

A consequence of the distinction between parsing and execution is that you can't simply stuff a command into a string and execute it. When you write ${grep_cmd}, only splitting and globbing happen, not parsing, so a character like | has no special meaning.

If you absolutely need to stuff a shell command into a string, you can eval it:

eval "$grep_cmd"

Note the double quotes — the value of the variable contains a shell command, so we need its exact string value. However, this approach tends to be complicated: you need to really have something in shell source syntax in there. If you need, say, a file name, then this file name must be properly quoted. So you can't just put $pattern and $@ in there, you need to build a string which when parsed results in a single word containing the pattern, and in a list of words containing the arguments.

To summarize: do not stuff shell commands into variables. Instead, use functions. If you need a simple command with arguments, and not something more complex such as a pipeline, you can use an array instead (an array variable stores a list of strings).

Here's one possible approach. The run_grep function isn't really needed with the code you've shown; I'm including it here on the assumption that this is a small part of a larger script and there's a lot more intermediate code. If this is really the whole script, just run grep at the point where you know what to pipe it into. I've also fixed the code that builds the filter, which didn't seem right (for example, . in a regexp means “any character”, but I think you want a literal dot).

grep_cmd=(grep -I -r "$pattern" "$@")

if (( ${#file_types[@]} > 0 )); then
    regexp='\.\('
    for file_type in "${file_types[@]}"; do
      regexp="$regexp$file_type\\|"
    done
    regexp="${regexp%?}):"
    run_grep () {
      "${grep_cmd[@]}" | grep "$file_types"
    }
else
  run_grep () {
    "${grep_cmd[@]}"
  }
fi

run_grep

¹ _{More generally, using the value of IFS.}
² _{For experts only: always use double quotes around variable and command substitutions, unless you understand why you leaving them off produces the right effect.}
³ _{For experts only: if you need to stuff a shell command into a variable, be very careful with the quoting.}

---

Note that what you're doing seems overly complex and not reliable — what if you have a file that contains foo.c: 42? GNU grep has a --include option to look only in certain files in a recursive traversal — just use it.

grep_cmd=(grep -I -r)
for file_type in "${file_types[@]}"; do
  grep_cmd+=(--include "*.$file_type")
done
"${grep_cmd[@]}" "$pattern" "$@"

Comments

[Admin]

It would be a lot simpler and safer if you just ran the command instead of storing it in $grep_cmd and trying to run it later.

[Admin]

Please elaborate. I am such a noob.

[Admin]

This shows how to grep on a grep output. stackoverflow.com/questions/14735511/…

[DevWouter]

This is exactly what I had before someone told me to change it :)