How can I detect if I'm in a subshell?

Question

I'm trying to write a function to replace the functionality of the exit builtin to prevent myself from exiting the terminal.

I have attempted to use the SHLVL environment variable but it doesn't seem to change within subshells:

$ echo $SHLVL
1
$ ( echo $SHLVL )
1
$ bash -c 'echo $SHLVL'
2

My function is as follows:

exit () {
    if [[ $SHLVL -eq 1 ]]; then
        printf '%s\n' "Nice try!" >&2
    else
        command exit
    fi
}

---

This won't allow me to use exit within subshells though:

$ exit
Nice try!
$ (exit)
Nice try!

What is a good method to detect whether or not I am in a subshell?

Solution 1

In bash, you can compare $BASHPID to $$

$ ( if [ "$$" -eq "$BASHPID" ]; then echo not subshell; else echo subshell; fi )
subshell
$   if [ "$$" -eq "$BASHPID" ]; then echo not subshell; else echo subshell; fi
not subshell

If you're not in bash, $$ should remain the same in a subshell, so you'd need some other way of getting your actual process ID.

One way to get your actual pid is sh -c 'echo $PPID'. If you just put that in a plain ( … ) it may appear not to work, as your shell has optimized away the fork. Try extra no-op commands ( : ; sh -c 'echo $PPID'; : ) to make it think the subshell is too complicated to optimize away. Credit goes to John1024 on Stack Overflow for that approach.

Solution 2

How about BASH_SUBSHELL?

BASH_SUBSHELL
      Incremented by one within each subshell or subshell environment when the shell
      begins executing in that environment. The initial value is 0.

$ echo $BASH_SUBSHELL
0
$ (echo $BASH_SUBSHELL)
1

Solution 3

[this should've been a comment, but my comments tend to be deleted by moderators, so this will stay as an answer that I could use it as a reference even if deleted]

Using BASH_SUBSHELL is completely unreliable as it be only set to 1 in some subshells, not in all subshells.

$ (echo $BASH_SUBSHELL)
1
$ echo $BASH_SUBSHELL | cat
0

Before claiming that the subprocess a pipeline command is run in is not a really real subshell, consider this man bash snippet:

Each command in a pipeline is executed as a separate process (i.e., in a subshell).

and the practical implications -- it's whether a script fragment is run a subprocess or not which is essential, not some terminology quibble.

The only solution, as already explained in the answers to this question is to check whether $BASHPID equals $$ or, portably but much less efficient:

if [ "$(exec sh -c 'echo "$PPID"')" != "$$" ]; then
    echo you\'re in a subshell
fi

Comments

[K7AAY]

stackoverflow.com/questions/4511407/…

[kemotep]

I am no expert but quickly looking things up it looks like you are already doing things correctly. $SHLVL keeps track of what level you are at. anything more than 1 would be a subshell.

[jesse_b]

@K7AAY: Yeah that's where I got the SHLVL idea from but unfortunately it doesn't work from a subshell only a new bash invocation. @kemotep if you look at the example at the top of my question you can see that SHLVL in fact does not work.

[mosvy]

Possible duplicate of How can I get the pid of a subshell?

[Sparhawk]

@mosvy I feel like that is a different question. e.g. the BASH_SUBSHELL answer (even if controversial) wouldn't apply to that question.

[user541686]

Saw the title on HNQ and thought this was a quantum mechanics question...

[Greg Burghardt]

I feel like I saw a movie about this once. Something like you need to keep an item with you at all times as an anchor... like a top. If you are in a sub shell, the top keeps spinning and never falls over. If you are in the top level shell it eventually stops and topples over.

[vijay]

@GregBurghardt Inception

[muru]

Nit: BASH_SUBSHELL is set pretty reliably, but getting its value correctly is iffy. Note what the docs say: "Incremented by one within each subshell or subshell environment when the shell begins executing in that environment." I think that in the pipe example, bash hasn't yet begun executing in that subshell when the variable is expanded. You can compare echo $BASH_VERSION with declare -p BASH_VERSION - the latter should reliably output 1 with pipes, background jobs, etc.

[mosvy]

@muru still feels like a bug. if it hadn't begun executing yet then how come that $BASHPID already has the right value?

[muru]

Even say, eval 'echo $BASH_SUBSHELL $BASHPID' | cat will output 1 for BASH_SUBSHELL, because the variable is expanded after execution has started.

[mosvy]

all those arguments should also apply to to process & commands substitution, bg processes, yet it's only the pipelines which are different. Looking at the code, incrementing subshell_level really is deferred in the case of foreground pipelines, which probably has some reason, but which I'm not able to make out ;-)

[muru]

You're right. Seems Chet explicitly intends it that way. lists.gnu.org/archive/html/bug-bash/2015-06/msg00050.html : "BASH_SUBSHELL measures (...) subshells, not pipeline elements." lists.gnu.org/archive/html/bug-bash/2015-06/msg00054.html: "I'm going to think about whether I should document the status quo or expand the definition of `subshell' that $BASH_SUBSHELL reflects."

[JoL]

The echo does run in a separate process, but the expansion of $BASH_SUBSHELL in your example doesn't. Bash has no need of a subshell there, because it's just one command it needs to exec. It doesn't need to run more shell-code after that. I would argue that the (i.e., in a subshell) you quoted from the manpage is a mistake in the documentation. They probably meant e.g. instead of i.e.. As an example of a pipe that does imply a subshell, for i in 1; do echo $BASH_SUBSHELL; done | cat outputs 1.

[Eric Duminil]

It would have been a convenient command in the movie Inception.

[mosvy]

@JoL you're wrong, the expansion happens in the separate process too, please read the links and examples from this discussion above; or just try with echo $$ $BASHPID $BASH_SUBSHELL | cat.

[Granny Aching]

In Inception it's probably $SHLVL