bash script executed over ssh returns incorrect exit code 0
Solution 1
I am able to duplicate this using the command you used, and I am able to resolve it by wrapping the remote command in quotes. Here are my test cases:
#!/bin/bash -x
echo 'Unquoted Test:'
ssh evil sh -x -c exit 5 && echo OK || echo FAIL
echo 'Quoted Test 1:'
ssh evil sh -x -c 'exit 5' && echo OK || echo FAIL
echo 'Quoted Test 2:'
ssh evil 'sh -x -c "exit 5"' && echo OK || echo FAIL
Here are the results:
bash-[540]$ bash -x test.sh
+ echo 'Unquoted Test:'
Unquoted Test:
+ ssh evil sh -x -c exit 5
+ exit
+ echo OK
OK
+ echo 'Quoted Test 1:'
Quoted Test 1:
+ ssh evil sh -x -c 'exit 5'
+ exit
+ echo OK
OK
+ echo 'Quoted Test 2:'
Quoted Test 2:
+ ssh evil 'sh -x -c "exit 5"'
+ exit 5
+ echo FAIL
FAIL
In the first test and second tests, it seems the 5
is not being passed to exit
as we would expect it to be. It just seems to be disappearing. It's not going to exit
, sh
isn't complaining about 5: command not found
, and ssh
isn't complaining about it.
In the third test, exit 5
is quoted within the larger command to run on the remote host, same as in the second test. This ensures that the 5
is passed to exit
, and both are executed as the -c
option to sh
. The difference between the second and third tests is that the whole set of commands and arguments is sent to the remote host quoted as a single command argument to ssh
.
Solution 2
As noted in the answer you already have, the remote sh
is not executing exit 5
. Just exit
:
$ ssh test sh -x -c 'exit 5'; echo $?
+ exit
0
What is happening here is explained, for instance, in this answer:
ssh
executes a remote shell and passes a string to it, not a list of arguments.
When we execute ssh host sh -c 'exit 5'
:
- The local shell removes the single quotes (quote removal);
- The
ssh
client gets the argumentshost
,sh
,-c
, andexit 5
. It concatenates them to a string and sends it to the remote host; - On the remote host,
ssh
invokes a shell and passes it the stringsh -c exit 5
; - The remote shell invokes
sh
and passes it the-c
option,exit
as the command string, and5
as the command name.
Note that, if we add words after exit 5
, they are just passed to sh
as further arguments - no error related to them not being recognized by the shell:
$ ssh test sh -x -c 'exit 5' a b c; echo $?
+ exit
0
strace
confirms that 5
is not part of the command string given to sh
, here; it is an argument:
$ ssh test strace -e execve sh -c 'exit 5'; echo $?
execve("/usr/bin/sh", ["sh", "-c", "exit", "5"], 0x7ffc0d744c38 /* 14 vars */) = 0
+++ exited with 0 +++
0
In order to execute sh -c 'command'
on a remote host as intended, we have to be sure to properly send it the quotes too:
$ ssh test "sh -x -c 'exit 5'"; echo $?
+ exit 5
5
To make it clear that quoting the whole remote command is not relevant to our current issue, we could just write:
$ ssh test sh -x -c "'exit 5'"; echo $?
+ exit 5
5
Escaping the inner quotes with backslashes, instead of quoting two times, would work as well.
A note about the command ssh host sh -c ':; exit 5'
(from the comments to your question). What it does is:
$ ssh test sh -x -c ':; exit 5'; echo $?
+ :
5
That is, exit 5
is executed by the outer shell, not by sh
. Again, to let sh
exit with the desired code:
$ ssh test sh -x -c "':; exit 5'"; echo $?
+ :
+ exit 5
5
Solution 3
The other answers are good at answering the question in lieu of the examples given. My real-world application is more complicated and involves a series of scripts and sub-processes. Here is a boiled-down example script I want to execute:
#!/bin/bash
sub-process-that-fails
# store and echo returncode for debug purposes
rc=$?
echo $rc
exit $rc
Trying to make sure that the remotely executed shell was actually bash and not dash (as pointed out by @JeffSchaller), I tried calling the script like this:
~$ ssh -t -t host /bin/bash -x /srv/scripts/run.sh ; echo $?
Which led to this weird output:
+ sub-process-that-fails
+ rc=5
+ echo 5
5
+ exit 5
0
After hours of poking around, I noticed there was a trap 'kill 0' EXIT
set in the .bashrc
. This is done to kill all sub-processes in case bash is killed. bash's trace does not seem to display this trap's execution. I moved the trap into the wrapper script. Now I can see what actually is executed:
+ trap 'kill 0' EXIT
+ sub-process-that-fails
+ rc=5
5
+ echo 5
+ exit 5
+ kill 0
0
The remote shell exits with the last command's exit code. It's kill 0
and it exits with 0.
Related videos on Youtube
Hermann
I am grateful for every day that I do not need to use a computer.
Updated on September 18, 2022Comments
-
Hermann almost 2 years
I am trying to automate a process which involves running scripts on various machines via ssh. It is vital to capture both output and the return code (for the detection of errors).
Setting the exit code explicitly works as expected:
~$ ssh host exit 5 && echo OK || echo FAIL FAIL
However, if there is a shell script signalling an unclean exit, ssh always returns 0 (script simulated by string execution):
~$ ssh host sh -c 'exit 5' && echo OK || echo FAIL OK
Running the very same script on the host in an interactive shell works just fine:
~$ sh -c 'exit 5' && echo OK || echo FAIL FAIL
I am confused as to why this happens. How can I tell ssh to propagate bash's return code? I may not change the remote scripts.
I am using public key authentication, the private key is unlocked – there is no need for user interaction. All systems are Ubuntu 18.04. Application versions are:
OpenSSH_7.6p1 Ubuntu-4ubuntu0.1, OpenSSL 1.0.2n 7 Dec 2017
GNU bash, Version 4.4.19(1)-release (x86_64-pc-linux-gnu)
Note: This question is different from these seemingly similar questions:
- bash shell - ssh remote script capture output and exit code?
- https://stackoverflow.com/questions/15390978/shell-script-ssh-command-exit-status
- https://stackoverflow.com/questions/36726995/exit-code-from-ssh-command
- https://superuser.com/questions/652729/command-executed-via-ssh-does-not-return-proper-return-code
-
Jeff Schaller over 5 yearsAre you certain that "sh" on the remote system is bash?
-
Jeff Schaller over 5 yearsIt seems like there was a similar problem earlier, where the shell doesn't fork because there's a simple command. Does the behavior change if you ask it to run
sh -c 'sleep 0.1; exit 5'
? -
Hermann over 5 years@JeffSchaller No, I am not certain. On Ubuntu,
/bin/sh
actually points to/bin/dash
. Nevertheless the behaviour does not change if I use absolute paths (/bin/bash
and/bin/sh
instead ofsh
). I hope there is no further auto-redirect going on. -
Hermann over 5 years@JeffSchaller Yes, even
ssh host sh -c ':; exit 5'
yields the expected return code. This does not help me as in the real world script, there is a lot more going on. I want to examine what forks where and then improve the examples in my question.
-
Hermann over 5 yearsIt does not help me in actually solving the problem I encounter, but I am accepting this answer as it shortly explains what is going on.
-
Hermann over 5 yearsThis answer adds detail to the one already given. Unfortunately, I can only officially "accept" one answer. The behaviour still feels non-intuitive to me.
-
fra-san over 5 years@Hermann No problem with the accept part, of course. Feel free to further point out what you feel less intuitive, answers can always be extended/improved.
-
Hermann over 5 yearsThe answer is fine. It is just the combination of ssh itself not needing quotes, but double quoting being a necessity as quotes are considered by both, local and remote shell.
-
Tim Kennedy over 5 yearsYeah. I would really love to know there the
5
is going in the first two tests. Especially the second where it's quoted to one level. I guess the remote shell is eating those quotes before it gets tosh
. But I still don't get where the5
is going. If I runsh -c echo 5
, the 5 is nowhere to be seen, either. Butsh -c 'echo 5'
does what you expect and echoes5
. -
Tim Kennedy over 5 yearsA very informative use of
strace
. Any idea what's actually happening to the5
? None of the shells seems to be doing anything with the5
without the double layer of quotes. -
ilkkachu over 5 yearsAlso, if you just remove the
trap
, it works as expected, right? Similarly tossh $somehost /bin/false
setting the (client-side) exit status to1
? -
fra-san over 5 years@TimKennedy The
5
(and what follows, if you add other words) are the zeroth and following parameters tosh
. They are just unused here, except for the zeroth one that is used as the command name - but giving the command a name is probably not what we want. You could actually use them, and you can see them, for example, changingexit
inssh test sh -x -c exit 5
into\'echo \"\$0\" \"\$@\" \'
, as inssh test sh -x -c \'echo \"\$0\" \"\$@\" \' 5 a b c
. The whole point is that, without proper quoting, the command string wanted bysh -c
ends up being just the first word that follows. -
Hermann over 5 yearsYes. But the
trap
is there for a reason. Working around that is not the scope of this question, though. -
Tim Kennedy over 5 yearsi guess I'm not used to seeing arguments just disappear unused. your explanation clears it up, though.
-
ilkkachu over 5 years@TimKennedy, in the same way, just running
sh -c "echo foo" a b c
would ignore the extra arguments. The script (given as an argument to-c
) just doesn't use them.