Trapping errors in command substitution using "-o errtrace" (ie set -E)

Question

According to this ref manual:

-E (also -o errtrace)

If set, any trap on ERR is inherited by shell functions, command substitutions, and commands executed in a subshell environment. The ERR trap is normally not inherited in such cases.

However, I must be interpreting it wrongly, because the following does not work:

#!/usr/bin/env bash
# -*- bash -*-

set -e -o pipefail -o errtrace -o functrace

function boom {
  echo "err status: $?"
  exit $?
}
trap boom ERR


echo $( made up name )
echo "  ! should not be reached ! "

I already know simple assignment, my_var=$(made_up_name), will exit the script with set -e (ie errexit).

Is -E/-o errtrace supposed to work like the above code? Or, most likely, I misread it?

Solution 1

Note: zsh will complain about "bad patterns" if you don't configure it to accept "inline comments" for most of the examples here and don't run them through a proxy shell as I have done with sh <<-\CMD.

Ok, so, as I stated in the comments above, I don't know specifically about bash's set -E, but I know that POSIX compatible shells provide a simple means of testing a value if you desire it:

    sh -evx <<-\CMD
    _test() { echo $( ${empty:?error string} ) &&\
        echo "echo still works" 
    }
    _test && echo "_test doesnt fail"
    # END
    CMD
sh: line 1: empty: error string
+ echo

+ echo 'echo still works'
echo still works
+ echo '_test doesnt fail'
_test doesnt fail

Above you'll see that though I used parameter expansion to test ${empty?} _test() still returns a pass - as is evinced in the last echo This occurs because the failed value kills the $( command substitution ) subshell that contains it, but its parent shell - _test at this time - keeps on trucking. And echo doesn't care - it's plenty happy to serve only a \newline; echo is not a test.

But consider this:

    sh -evx <<-\CMD
    _test() { echo $( ${empty:?error string} ) &&\
            echo "echo still works" ; } 2<<-INIT
            ${empty?function doesnt run}
    INIT
    _test ||\
            echo "this doesnt even print"
    # END
    CMD
_test+ sh: line 1: empty: function doesnt run

Because I fed _test()'s input with a pre-evaluated parameter in the INIT here-document now the _test() function doesn't even attempt to run at all. What's more the sh shell apparently gives up the ghost entirely and echo "this doesnt even print" doesn't even print.

Probably that is not what you want.

This happens because the ${var?} style parameter-expansion is designed to quit the shell in the event of a missing parameter, it works like this:

${parameter:?[word]}

Indicate Error if Null or Unset. If parameter is unset or null, the expansion of word (or a message indicating it is unset if word is omitted) shall be written to standard error and the shell exits with a non-zero exit status. Otherwise, the value of parameter shall be substituted. An interactive shell need not exit.

I won't copy/paste the entire document, but if you want a failure for a set but null value you use the form:

${var :? error message }

With the :colon as above. If you want a null value to succeed, just omit the colon. You can also negate it and fail only for set values, as I'll show in a moment.

Another run of _test():

    sh <<-\CMD
    _test() { echo $( ${empty:?error string} ) &&\
            echo "echo still works" ; } 2<<-INIT
            ${empty?function doesnt run}
    INIT
    echo "this runs" |\
        ( _test ; echo "this doesnt" ) ||\
            echo "now it prints"
    # END
    CMD
this runs
sh: line 1: empty: function doesnt run
now it prints

This works with all kinds of quick tests, but above you'll see that _test(), run from the middle of the pipeline fails, and in fact its containing command list subshell fails entirely, as none of the commands within the function run nor the following echo run at all, though it is also shown that it can easily be tested because echo "now it prints" now prints.

The devil is in the details, I guess. In the above case, the shell that exits is not the script's _main | logic | pipeline but the ( subshell in which we ${test?} ) || so a little sandboxing is called for.

And it may not be obvious, but if you wanted to only pass for the opposite case, or only set= values, it's fairly simple as well:

    sh <<-\CMD
    N= #N is NULL
    _test=$N #_test is also NULL and
    v="something you would rather do without"    
    ( #this subshell dies
        echo "v is ${v+set}: and its value is ${v:+not NULL}"
        echo "So this ${_test:-"\$_test:="} will equal ${_test:="$v"}"
        ${_test:+${N:?so you test for it with a little nesting}}
        echo "sure wish we could do some other things"
    )
    ( #this subshell does some other things 
        unset v #to ensure it is definitely unset
        echo "But here v is ${v-unset}: ${v:+you certainly wont see this}"
        echo "So this ${_test:-"\$_test:="} will equal NULL ${_test:="$v"}"
        ${_test:+${N:?is never substituted}}
        echo "so now we can do some other things" 
    )
    #and even though we set _test and unset v in the subshell
    echo "_test is still ${_test:-"NULL"} and ${v:+"v is still $v"}"
    # END
    CMD
v is set: and its value is not NULL
So this $_test:= will equal something you would rather do without
sh: line 7: N: so you test for it with a little nesting
But here v is unset:
So this $_test:= will equal NULL
so now we can do some other things
_test is still NULL and v is still something you would rather do without

The above example takes advantage of all 4 forms of POSIX parameter substitution and their various :colon null or not null tests. There is more information in the link above, and here it is again.

And I guess we should show our _test function work, too, right? We just declare empty=something as a parameter to our function (or any time beforehand):

    sh <<-\CMD
    _test() { echo $( echo ${empty:?error string} ) &&\
            echo "echo still works" ; } 2<<-INIT
            ${empty?tested as a pass before function runs}
    INIT
    echo "this runs" >&2 |\
        ( empty=not_empty _test ; echo "yay! I print now!" ) ||\
            echo "suspiciously quiet"
    # END
    CMD
this runs
not_empty
echo still works
yay! I print now!

It should be noted that this evaluation stands alone - it requires no additional test to fail. A couple more examples:

    sh <<-\CMD
    empty= 
    ${empty?null, no colon, no failure}
    unset empty
    echo "${empty?this is stderr} this is not"
    # END
    CMD
sh: line 3: empty: this is stderr

    sh <<-\CMD
    _input_fn() { set -- "$@" #redundant
            echo ${*?WHERES MY DATA?}
            #echo is not necessary though
            shift #sure hope we have more than $1 parameter
            : ${*?WHERES MY DATA?} #: do nothing, gracefully
    }
    _input_fn heres some stuff
    _input_fn one #here
    # shell dies - third try doesnt run
    _input_fn you there?
    # END
    CMD
heres some stuff
one
sh: line :5 *: WHERES MY DATA?

And so finally we come back to the original question : how to handle errors in a $(command substitution) subshell? The truth is - there are two ways, but neither is direct. The core of the problem is the shell's evaluation process - shell expansions (including $(command substitution)) happen earlier in the shell's evaluation process than does current shell command execution - which is when your errors could be caught and trapped.

The issue the op experiences is that by the time the current shell evaluates for errors, the $(command substitution) subshell has already been substituted away - no errors remain.

So what are the two ways? Either you do it explicitly within the $(command substitution) subshell with tests as you would without it, or you absorb its results into a current shell variable and test its value.

Method 1:

    echo "$(madeup && echo \: || echo '${fail:?die}')" |\
          . /dev/stdin

sh: command not found: madeup
/dev/stdin:1: fail: die

    echo $?

126

Method 2:

    var="$(madeup)" ; echo "${var:?die} still not stderr"

sh: command not found: madeup
sh: var: die

    echo $?

1

This will fail regardless of the number of variables declared per line:

   v1="$(madeup)" v2="$(ls)" ; echo "${v1:?}" "${v2:?}"

sh: command not found: madeup
sh: v1: parameter not set

And our return value remains constant:

    echo $?
1

NOW THE TRAP:

    trap 'printf %s\\n trap resurrects shell!' ERR
    v1="$(madeup)" v2="$(printf %s\\n shown after trap)"
    echo "${v1:?#1 - still stderr}" "${v2:?invisible}"

sh: command not found: madeup
sh: v1: #1 - still stderr
trap
resurrects
shell!
shown
after
trap

    echo $?
0

Solution 2

If set, any trap on ERR is inherited by shell functions, command substitutions, and commands executed in a subshell environment

In your script its a command execution(echo $( made up name )). In bash commands are delimited with either ; or with new line. In the command

echo $( made up name )

$( made up name ) is considered as a part of the command. Even if this part fails and return with error, the whole command executes successfully as echo don't know about it. As the command return with 0 no trap triggered.

You need to put it in two commands, assignment and echo

var=$(made_up_name)
echo $var

Solution 3

This is due to a bug in bash. During the Command Substitution step in

echo $( made up name )

made runs (or fails to be found) in a subshell, but the subshell is "optimized" in such a way that it doesn't use some traps from the parent shell. This was fixed in version 4.4.5:

Under certain circumstances, a simple command is optimized to eliminate a fork, resulting in an EXIT trap not being executed.

With bash 4.4.5 or higher, you should see the following output:

error.sh: line 13: made: command not found
err status: 127
  ! should not be reached !

The trap handler has been called as expected, then the subshell exits. (set -e only causes the subshell to exit, not the parent, so that "should not be reached" message should, in fact, be reached.)

A workaround for older versions is to force the creation of a full, non-optimized subshell:

echo $( ( made up name ) )

The extra spaces are required to distinguish from Arithmetic Expansion.

Comments

[mikeserv]

echo $var doesn't fail - $var is expanded to empty before ever echo actually gets a look at it.

[mikeserv]

For instance, practically his exact specification: echo ${v=$( madeupname )} ; echo "${v:?trap1st} ! is not reached !"

[Admin]

To summarize: These parameter expansions are a great way to control if variables are set etc. In respect of the exit status of echo, I noticed that echo "abc" "${v1:?}" doesn't seem to execute(abc never printed). And the shell returns 1. This is true with or without a command even("${v1:?}" directly on the cli). But for OP's script, all that was required to trigger the trap is placing his variable assignment containing a substitution for a nonexistent command alone on a single line. Otherwise the behavior of echo is to return 0 always, unless interrupted like with the tests you explained.

[Admin]

Just v=$( madeup ). I don't see what is unsafe with that. It's just an assignment, the person mispelled the command for instance v="$(lss)". It errors. Yes you can verify with the error status of the last command $? - because it is the command on the line (an assignment with no command name) and nothing else - not an argument to echo. Plus, here it's trapped by the function as !=0, so you get feedback twice. Otherwise surely as you explain there is a better way to do this orderly within a framework but OP had 1 single line: an echo plus his failed substitution. He was wondering about echo.

[mikeserv]

Yes, simple assignment is mentioned in the question as well as a for granted, and while i couldnt offer specific advice on how to make use of -E, i did try to show similar results to the demonstrated issue were more than possible without resorting to bashisms. In any case, it is specifically the issues you mention - like single assignment per line - that make such solutions difficult to handle in a pipeline, which i also demonstrated how to handle. Though it's true what you say - there is nothing unsafe about simply assigning it.

[mikeserv]

Good point - perhaps more effort is called for. Ill think on it.