Why does SSH hang at the end of these commands and how can I make it exit?

Question

I run this:

ssh -t -vvv -i ~/.ssh/druid-keypair -o StrictHostKeyChecking=no ubuntu@${INSTANCE_ADDRESS} <<EOI

# Setup Oracle Java
...

# Install dependencies - mysql must be built from source, as the 12.04 apt-get hangs
export DEBIAN_FRONTEND=noninteractive
sudo debconf-set-selections <<< 'mysql-server-5.5 mysql-server/root_password password diurd'
sudo debconf-set-selections <<< 'mysql-server-5.5 mysql-server/root_password_again password diurd'
sudo apt-get -q -y -V --force-yes --reinstall install mysql-server-5.5

echo "ALL DONE with druid environment setup!"
exit
EOI

Note: I have tried with and without -t in ssh.

The debug output from -vvv is this:

...
ldconfig deferred processing now taking place
ALL DONE with druid environment setup!
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0

And then the process just sits there forever. Why won't the ssh command end? I have tried with -t and without, and I have tried with the exit and without. It doesn't make a difference :(

Update: When I type 'jobs' at the end of the script, I see:

JOBS:
[1]-  Running                 nohup bin/zookeeper-server-start.sh config/zookeeper.properties 2>&1 > /dev/null &
[2]+  Running                 nohup bin/kafka-server-start.sh config/server.properties 2>&1 > /dev/null &

How can I run these services and still have an ssh session that ends?

Update: I now manually disown these processes. The thing still doesn't exit. WTF mate?

Update: When executing line by line, two commands don't return to shell without hitting CR:

nohup bin/zookeeper-server-start.sh config/zookeeper.properties &
nohup bin/kafka-server-start.sh config/server.properties &

Solution

Typically, SSH terminal sessions hang if there are still background connections still open. By background connections, I mean things such as:

• X11 window forwarding
• STDOUT and STDERR

Have a look at the connections that are still active on your hung SSH session by typing ~# in your hung SSH terminal.

It could be that your script is opening sessions that you didn't realize. Or your remote machine's terminal configs like .profile (or .bashrc, etc.) may have something in it that establishes a session. Good luck hunting!

By the way, some of the other escape sequences offered by OpenSSH clients may also be useful:

Supported escape sequences:
  ~.  - terminate connection (and any multiplexed sessions)
  ~B  - send a BREAK to the remote system
  ~C  - open a command line
  ~R  - Request rekey (SSH protocol 2 only)
  ~^Z - suspend ssh
  ~#  - list forwarded connections
  ~&  - background ssh (when waiting for connections to terminate)
  ~?  - this message
  ~~  - send the escape character by typing it twice
(Note that escapes are only recognized immediately after newline.)

One other thing, if you want your SSH to just run your commands and immediately exit -- that is, you don't want a remote terminal session -- you can use the -f option to ssh. That will force the SSH connection to be a background job.

Comments

[Jingguo Yao]

-f works for me.

[Kostas]

I heard of -n for the same purpose but that didnt work for me!

[the_prole]

Does a background process count as a background connection and can one disown the background process so the ssh connection doesn't hang?

[FantomX1]

if by chance anyone used this in connection with sshpass , it is necessary to pass the -f parameter to the ssh command itself of course and not to the sshpass command