Passing pipeline value as parameter to xargs for use by eval echo

Question

I have a text file that I'm using as a template, it looks like this:

Hostname     : $HOSTNAME
Host Address : $HOSTADDRESS

My bash script sets two variables, HOSTNAME and HOSTADDRESS, reads the template file then does an eval to expand $HOSTNAME and $HOSTADDRESS:

HOSTNAME="SH_SQL_0089"
HOSTADDRESS="172.16.3.44"
TEMPLATE=`cat template.txt`
MESSAGE=`eval echo $TEMPLATE`

The resultant value of MESSAGE is:

Hostname     : SH_SQL_0089
Host Address : 172.16.3.44

Is it possible to reduce the last two lines to something like:

MESSAGE=$(cat template.txt | eval echo ????)

I tried using xargs:

MESSAGE=$( cat template.txt | xargs -i bash -c "eval echo {}" )

But it only substitutes $HOSTNAME and strips the carriage returns.

The reason I want to do this is that I need to add a third processing step and pipe the eval'd output to that. I feel I'm so close.

Solution 1

First, xargs can't work here because the substitutions would be executed in a subprocess (the bash processes that xargs launches). But only environment variables are passed to subprocesses, not shell variables. Evidently HOSTNAME was already in your script's environment when it started, but HOSTADDRESS isn't. At this point you might be tempted to export all variables, but even then xargs isn't a good solution because it suffers from numerous quoting issues − if your template contains \"' or if you want to retain whitespace, you're toast.

Now, looking at your current code: MESSAGE=`eval echo $TEMPLATE` is a complex way of writing eval MESSAGE=$TEMPLATE, except for quoting issues. And you do have quoting issues; for example you'll have noticed that all your whitespace has been collapsed. You've heard of Bobby Tables, haven't you? Shell expansion rules are pretty complex, but there are some rules that'll keep you sane:

• Always double quotes around variable substitutions "$foo" and command substitutions "$(bar)". You may violate this rule if you understand why you need to omit the quotes.
• Use $(…) rather than `…` for command substitutions. Quoting inside the backquote form is arcane and not portable, whereas quoting inside $(…) works normally.
• Use eval only if pressed at gunpoint. If you do, be very careful, and make it as simple as possible.

So what can go wrong with eval MESSAGE="$TEMPLATE"? It makes the shell evaluate

MESSAGE=Hostname     : $HOSTNAME
Host Address : $HOSTADDRESS

Oops, we need quotes around the part that should be the value of MESSAGE. The quotes have to get through to eval, so they need to go literally through the first stage of shell expansion: eval MESSAGE="\"$TEMPLATE\"".

MESSAGE="Hostname     : $HOSTNAME
Host Address : $HOSTADDRESS"

Better, but now you have exactly the Bobby Tables injection pattern — what if the template contains a quote? Then you need to escape the quotes. The four characters that have a special meaning between double quotes are \"$`, and you want $ to retain its meaning, so add a backslash before the other three.

TEMPLATE=$(sed -e 's/[\\"`]/\\&/g' <template.txt)
eval MESSAGE="\"$TEMPLATE\""

Now the shell will evaluate

MESSAGE="Hostname     : $HOSTNAME
Host Address : $HOSTADDRESS
Name         : Bobby \"drop\" O'Tables"

and all's good.

Note that proper quoting here is to protect against accidental parsing problems; whoever controls the template still has shell access (with $(hello)).

If you wanted the template included in your shell script, this would be naturally done with a heredoc.

MESSAGE=$(cat <<EOF)
Hostname     : $HOSTNAME
Host Address : $HOSTADDRESS
EOF

But with an external template you'd need to do two steps of evaluation, hence use eval, and bash's parsing is quite buggy when it comes to funky stuff like heredocs inside eval. There's surely a way that works, at least with bash 4, but I don't recommend risking it.

Solution 2

You want the shell to apply substitutions of variables, which means that the template text must be read by the shell itself as part of a command, but the template is also multi-line text so it's a bit tricky to get into the line-oriented processing normally done by UNIX commands.

One approach would be to use bash here-documents (all commands shown as typed at the shell prompt):

1. create the template file:

   $ cat template.txt
   Hi, $NAME
   
   Welcome to $PLACE
   

2. export the variables that should be substituted in the template text:

   $ export NAME=Bob
   $ export PLACE='unix&linux'
   

3. create a shell variable that holds a single "newline" \n character; in bash the easiest way is to simply open a string, type a newline and close it:

   $ newline='
   > '
   

4. finally, call bash to do the substitution:

   $ bash -c "cat <<__EOF__${newline}$(cat template.txt)${newline}__EOF__"
   Hi, Bob
   
   Welcome to unix&linux
   

Why does this work? Let's deconstruct it from the bottom: you're asking bash to execute a command (by the -c option), but the variable substitution ${newline} and the command expansion $(cat ...) happen in the parent shell before bash -c ... is actually executed. So the result is that bash -c sees a command string that embeds newlines and the whole template text. It's as if you had typed the following at a bash prompt:

cat <<__EOF__
...contents of template.txt...
__EOF__

Also note that the substitution variables need to be exported, since the variables are assigned in the parent shell but it's the "child" bash that is doing the substitution.

Note, however, that this approach can easily lead to quoting issues: since the template text is interpreted by the bash shell, backtics are expanded, which might result in commands being executed on your system.

Comments

[heyman]

I think you meant $(eval echo $(cat ... - but that would collapse the text into a single line.

[Gilles 'SO- stop being evil']

I'm not sure what you meant, but it's not going to help Kev write working code.

[MSpike]

Gilles: I think my solution is working, with the echo mentioned by Riccardo, and I think it was the answer for his question... Anyway she/he got a solution already :)

[kta]

How about export $(cat .env | xargs)?

[Gilles 'SO- stop being evil']

@kta I have no idea what you're trying to do with this command. What's .env? Why are you using cat? What's the point of invoking xargs without a command?