Can I get the exit code of a command executed in a subshell via ssh?

Question

I'm trying to use Paramiko to write a deployment script, and I'm having trouble with exit codes from the commands I run. I'm using code similar to that in this answer, but it's a little more complicated. Basically, from our dev boxes we have to go through a jump server, and from there to a series of production machines. Once there we have to switch to a system user (sudo su - systemuser) and then we can run commands.

The problem is that as I understand it I have 3 subshells - the ssh session, the nested ssh command and then the su subshell. I can't get Paramiko to give me back the exit code of the commands in the inner subshell - I guess the exit code it will eventually return will be that of the ssh command. I suspect this problem is not actually specific to Paramiko - does the SSH protocol even support this kind of usage?

I'm currently always executing:

(my command); echo "Process terminated with exit code $?"

and then parsing this out on the client, but it's pretty ugly - is there a nicer way?

Solution 1

I guess the exit code it will eventually return will be that of the ssh command. I suspect this problem is not actually specific to Paramiko - does the SSH protocol even support this kind of usage?

Yes, and no.

The exit status is always returned, but it may not be from where you think. Once you call invoke_shell(), your remote process is a shell, probably bash. The exit_status you receive will be the one from that shell, not any of the processes it ran. Paramiko (nor any ssh implementation) can ever return the exit status of your commands, because the exit statuses are never seen outside of that remote shell. It's up to you to determine what your exit status should be, and exit the shell with exit $EXIT_STATUS (often exit $? is enough)

If you can break up your commands, or wrap them in a script, then use the exec_command() method, you will have access to the correct exit status directly.

Alternatively, you can use shell command exec in the remote shell, and the shell's process will be replaced by the command called by exec, and its exit status will be returned.

All three of these methods will set the channel.exit_status attribute as expected.

Solution 2

I'm having a problem where the man page for 'ssh' says that it returns the exit code of the process it runs, but I can't seem to get it to return a non-zero error code. From the ssh man page:

 The session terminates when the command or shell on the remote machine
 exits and all X11 and TCP/IP connections have been closed.  The exit sta‐
 tus of the remote program is returned as the exit status of ssh.

That doesn't seem to be true.

But I would try something like this and see what happens on your system.

% ssh localhost bash -c "exit 3" ; echo $?
0

When I run a similar command locally, bash returns an exit code.

% bash -c 'exit 3' ; echo $?
3

The double quotes will be removed before ssh seems the commands however. So let's try more quotes.

% ssh localhost bash -c "'exit 3'" ; echo $?
3

Bingo. The "exit 3" was turning into "exit" followed by an ignored word on the bash command line. So bash was running the command "exit".

Unfortunately for me, I think this whole answer is a digression from the original questions and doesn't contain enough merit as a question in its own right. So thanks everyone for helping me answer by secondary question (not related to the original question).