Is this a bug in bash? `return` doesn't quit function if called from a pipe

bash shell-script pipe subshell

10,658

Solution 1

It's not a bug that you cannot exit a script or return from a function by exit or return in subshells. They are executed in another process and not affecting the main process.

Besides that, I suppose you are seeing undocumented behaviors of bash on (probably) undefined spec. In a function, no errors are asserted for return at top level of subshell commands and it just behaves like exit.

IMHO it's a bash bug for the inconsistent behavior of return depending on whether the main statement is in a function or not.

#!/bin/bash

o() {
    # Runtime error, but no errors are asserted,
    # each $? is set to the return code.
    echo | return 10
    echo $?
    (return 11)
    echo $?

    # Valid, each $? is set to the exit code.
    echo | exit 12
    echo $?
    (exit 13)
    echo $?
}
o

# Runtime errors are asserted, each $? is set to 1.
echo | return 20
echo $?
(return 21)
echo $?

# Valid, each $? is set to the exit code.
echo | exit 22
echo $?
(exit 23)
echo $?

Output:

$ bash script.sh 
10
11
12
13
script.sh: line 20: return: can only `return' from a function or sourced script
1
script.sh: line 22: return: can only `return' from a function or sourced script
1
22
23

Solution 2

It is not a bug in bash but its documented behavior:

Each command in a pipeline is executed in its own subshell

The return instruction is valid being inside a function definition but being in a subshell as well, it doesn't affect its parent shell so the next instruction, echo, is executed regardless. It is nevertheless a non portable shell construction as the POSIX standard allows the commands composing a pipeline to be executed either in a subshell (the default) or the top one (an allowed extension).

Additionally, each command of a multi-command pipeline is in a subshell environment; as an extension, however, any or all commands in a pipeline may be executed in the current environment. All other commands shall be executed in the current shell environment.

Hopefully, you can tell bash to behave the way your expect with a couple of options:

$ set +m # disable job control
$ shopt -s lastpipe # do not run the last command of a pipeline a subshell 
$ o(){ echo | while read -r; do return 0; done; echo $?;}
$ o
$          <- nothing is printed here

Solution 3

Per POSIX documentation, using return outside of function or sourced script is unspecified. So, it depends on your shell to handle.

SystemV shell will report error, while in ksh, return outside of function or sourced script behave like exit. Most other POSIX shells and schily's osh also behave like that:

$ for s in /bin/*sh /opt/schily/bin/osh; do
  printf '<%s>\n' $s
  $s -c '
    o(){ echo | while read l; do return 0; done; echo $?;}; o
  '
done
</bin/bash>
0
</bin/dash>
0
</bin/ksh>
</bin/lksh>
0
</bin/mksh>
0
</bin/pdksh>
0
</bin/posh>
0
</bin/sh>
0
</bin/yash>
0
</bin/zsh>
</opt/schily/bin/osh>
0

ksh and zsh didn't output because the last part of pipe in these shells was executed in current shell instead of subshell. The return statement affected the current shell environment which called the function, cause the function return immediately without printing anything.

In interactive session, bash only report the error but didn't terminated the shell, schily's osh reported the error and terminated the shell:

$ for s in /bin/*sh; do printf '<%s>\n' $s; $s -ci 'return 1; echo 1'; done
</bin/bash>
bash: return: can only `return' from a function or sourced script
1
</bin/dash>
</bin/ksh>
</bin/lksh>
</bin/mksh>
</bin/pdksh>
</bin/posh>
</bin/sh>
</bin/yash>
</bin/zsh>
</opt/schily/bin/osh>
$ cannot return when not in function

(zsh in interactive session and output is terminal do not terminated, bash, yash and schily's osh reported the error but didn't terminate the shell)

Solution 4

I think that you got the expected behavior, in bash, each command in a pipeline is executed in a subshell. You can convice yourself by trying to modify a global variable of your function:

foo(){ x=42; : | x=3; echo "x==$x";}

By the way, the return is working but it return from the subshell. Again you can check that:

foo(){ : | return 1; echo$?; echo "This should not be printed.";}

Will output the following:

1
This should not be printed.

So return statement correctly exited the subshell

Solution 5

The more general answer is that bash and some other shells normally put all the elements of a pipeline into separate processes. This is reasonable when the command line is

program₁ | program₂ | program₃

since programs are normally run in separate processes anyway (unless you say exec program). But it can come as a surprise for

command₁ | command₂ | command₃

where some or all of the commands are built-in commands. Trivial examples include:

$ a=0
$ echo | a=1
$ echo "$a"
0
$ cd /
$ echo | cd /tmp
$ pwd
/

A slightly more realistic example is

$ t=0
$ ps | while read pid rest_of_line
> do
>     : $((t+=pid))
> done
$ echo "$t"
0

where the entire while … do … done loop is put into a subprocess, and so its changes to t aren’t visible to the main shell after the loop ends. And that’s exactly what you’re doing — piping into a while loop, causing the loop to run as a subshell, and then trying to return from the subshell.

View more solutions

10,658

Rizwan Khan

Updated on September 18, 2022

Comments

Rizwan Khan over 1 year
I'm been having some weird problems with bash lately. While trying to simplify my script, I came up with this small piece of code:
```
$ o(){ echo | while read -r; do return 0; done; echo $?;}; o
0
$ o(){ echo | while read -r; do return 1; done; echo $?;}; o
1
```
return should have exited the function without printing $?, shouldn't it? Well, then I checked if I can return from a pipe alone:
```
$ echo | while read -r; do return 1; done
bash: return: can only `return' from a function or sourced script
```
The same happens without a while loop:
```
$ foo(){ : | return 1; echo "This should not be printed.";}
$ foo
This should not be printed.
```
Is there something I'm missing here? A Google search brought nothing about this! My bash version is 4.2.37(1)-release on Debian Wheezy.
- jlliagre over 8 years
  
  Anything wrong with the settings I suggested in my reply that allow your script to behave the intuitive way you expected it to do?
- Rizwan Khan over 8 years
  
  @jlliagre It is a rather complex script on the thousands of lines. With the concern of breaking something else, I prefer to just avoid running a pipe within the function, so I replaced it with a process substitution. Thanks!
- Rizwan Khan over 8 years
  
  @LightnessRacesinOrbit A while loop is a very common usage for a pipe with return. The second example is more straight to the point, but it is something I don't believe anyone would ever use...
- schily over 8 years
  
  Unfortunately my correct answer has been deleted... You are in a grey zone as you do something that is unspecified. The behavior depends on how the shell interprets pipes and this is even different between the Bourne Shell and the Korn Shell even though ksh was derived from sh sources. In the Bourne Shell, the while loop is in a subshell therefore you see the echo as with bash, In ksh the while loop is the foreground process and thus ksh does not call echo with your example.
- Lightness Races in Orbit over 8 years
  
  @schily: There is no deleted answer on this question.
- schily over 8 years
  
  Just because you cannot see it does not mean it dis not happen.
Richard Fortune over 8 years

The lack of error verbosity may be undocumented. But the fact that return doesn't work from a top-level command sequence in a subshell, and in particular doesn't exit the subshell, is what existing documents have already made me expect. The OP could use exit 1 || return 1 where they are trying to use return, and should then get the expected behavior. EDIT: @herbert's answer indicates that toplevel return in subshell is functioning as exit (but only from the subshell).
Richard Fortune over 8 years

Hence, to exit the function, use foo(){ : | return 1 || return 2; echo$?; echo "This should not be printed.";}; foo; echo $? and you'll get a result of 2. But for clarity I would make the return 1 be exit 1.
yaegashi over 8 years

@dubiousjim Updated my script. I mean return in a simple subshell sequence should be asserted as a runtime error in any cases, but actually it's not when it occurs in a fucntion. This problem has also been discussed in gnu.bash.bug, but there's no conclusion.
jlliagre over 8 years

It can be argued return is used inside a function here.
cuonglm over 8 years

@jlliagre: Not sure what you mean, return was use inside subshell inside function, except ksh and zsh.
Rizwan Khan over 8 years

Since return won't quit the function, wouldn't it make more sense if the shell just printed bash: return: can only `return' from a function or sourced script, instead of giving the user a false sense that the function might have returned?
jlliagre over 8 years

I mean being inside a subshell which is itself inside a function doesn't necessarily mean being outside that function, i.e. nothing in the standard states pipeline components are to be considered to be outside the function where they are located. This would deserve to be clarified by the Open Group.
jlliagre over 8 years

@TeresaeJunior That would be tricky to implement. The issue is return can arguably be considered to be inside the function, and that it might return from it depending on the context. You can only use this construction if you make sure you use the right shell and/or option combination.
cuonglm over 8 years

I think no. That's outside the function. The shell which called the function and the subshell which executed return are different.
cuonglm over 8 years

I don't see anywhere in documentation said that the return inside subshell is valid. I bet that this feature was copied from ksh, the return statement outside function or sourced script behave like exit. I'm not sure about the original Bourne shell.
Scott - Слава Україні over 8 years

@jlliagre: Maybe Teresa is confused about the terminology of what she's asking for, but I don't see why it would be "tricky" for bash to issue a diagnostic if you execute a return from a subshell. After all, it does know that it's in a subshell, as evidenced by the $BASH_SUBSHELL variable. The biggest problem is that this could lead to false positives; a user who understands how subshells work could have written scripts that use return in lieu of exit to terminate a subshell. (And, of course, there are valid cases where one might want to set variables or do a cd in a subshell.)
Rizwan Khan over 8 years

@Scott I think I understand the situation well. A pipe creates a subshell, and return is returning from the subshell instead of failing, since it is inside an actual function. The problem is that help return specifically states: Causes a function or sourced script to exit with the return value specified by N. From reading the documentation, any user would expect it to at least fail or print a warning, but never to behave like exit.
Scott - Слава Україні over 8 years

It seems to me that anybody who expects a return in a subshell in a function to return from the function (in the main shell process) doesn’t understand subshells very well. Conversely, I would expect a reader who understands subshells to expect return in a subshell in a function to terminate the subshell, just as exit would.
Incnis Mrsi over 8 years

By the way, is there some substantiation for the fact that all members of a pipeline (not all but one) are executed in subshells?
Scott - Слава Україні over 8 years

@IncnisMrsi: See jlliagre's answer.
jlliagre over 8 years

@Scott The fact is it is not obvious for many (most?) shell scripts writers that there are subshells here in the first place. It is quite unfortunate the committee that defined the POSIX shell standard didn't pick the ksh clever way as their model like they generally did for the remaining features and behavior. The counter-intuitive POSIX default behavior is the cause of countless bugs and ksh to bash porting issues.
jlliagre over 8 years

I understand your reasoning which rightly explains the issue, my point is according to shell grammar described in the POSIX standard, the pipeline is part of the compound-list which is part of the compound-command which is the body of the function. Nowhere is it stated pipeline components are to be considered outside the function. Just like if I am in a car and that car is parked in a garage, I can assume I am in that garage too ;-)
jlliagre over 8 years

@Scott again, Being in a subshell is not enough to assert calling return should be considered an error. After the fork, both the parent and the child shells are technically inside the same function although each one in their own instance, that is the reason why return is accepted as a valid instruction in the subshell.
schily over 8 years

Your answer is not correct as it is unspecified whether the while loop is in a subshell or is the foreground process. Regardless of the implementation of the actual shell, the return statement is in a function and thus legal. The resulting behavior however is unspecified.
schily over 8 years

Using shopt makes the script non-portaple. As mentioned in my correct by removed answer: The command from the OP is based on unspecified behavior and thus can just be seen as unreliable. Avoid read to the right side of a pipe as it may be in a subshell, avoid return from a complex construct as it may also be in a subshell.
schily over 8 years

Your second example is nice but not related to the code from the OP. The return statement from the OP was in a function. The reason why ksh does not print the exit code is because ksh makes the right side of a pipe the foreground process and because ksh does not execute a while loop in a subshell.
jlliagre over 8 years

@schily Read closer my reply. I definitely state that the construction is non portable and why. I suggested the shopt workaround because the question is not about Bourne style shells in general but very bash specific (have a look to question title, its content and its tags). Stackexchange sites are not forums: see meta.stackexchange.com/a/92110/182055
cuonglm over 8 years

@schily: Because the OP said This seems to happen only in interactive mode, so I made an example for it.
schily over 8 years

But you tested something completely different than what the OP reported. You tested retun with no function active while the OP had a problem where the return succeeded but just did something the OP did not expect.
cuonglm over 8 years

@schily: Did you read the question carefully? The OP shown using return in interactive mode, and output is terminal then bash reported bash: return: can only return' from a function or sourced script. My example just made it more details.
schily over 8 years

OK, so you did not read the text from the OP carefully enough. He was doing own research, but his concern was related to the fact that return did not fully leave the function in his case. He already discovered that his original problem is related to the pipe and seems to be interested in getting an explanation for this. I explained that this is a grey zone as the way pipes are implemented is not in the standard and that in order to give the impression of a successful "return", the right side of the pipe must be a foreground process.
cuonglm over 8 years

@schily: I didn't understand what do you mean? The first part of my answer is for the OP question, the second part is for making the OP's research more clear.
schily over 8 years

OK, then I understand why you added this test. The main message on the other side should be that because the original Bourne Shell did run the right side of a pipe in a subshell, we cannot reliably use echo bla | read ANS and expect $ANS to be present in the main shell process unless the shell is e.g. known to be a Korn Shell. I would prefer the right side to be always a foreground process.
Rizwan Khan over 8 years

@cuonglm I removed that edit yesterday. It actually happens in non-interactive mode too. Sorry for that!
cuonglm over 8 years

@downvoter: Why the downvote?
jlliagre over 8 years

You shouldn't write it is an undocumented behavior while the fact pipe components are in a subshell is documented in the bash manual page. You shouldn't write the behavior is probably based on undefined specs while POSIX specify the allowed behaviors. You shouldn't suspect a bash bug while bash is following the POSIX standard by allowing return in a function but not outside.