Keeping an SSH Tunnel open, and running a remote command

Question

A computer behind a firewall (at a remote location) connects to my server to open a tunnel... the intention is that from the server I can connect to the computer over SSH.

The computer makes the connection with:

while true; do
    PORT=$(((RANDOM %= 1000) + 2000));
    ssh -R $PORT:localhost:22 -o ServerAliveInterval=30 "server" "record_port.sh '$PORT';";
    sleep 5;
done

The "record_port.sh" shell script on the server contains:

echo "$PORT" > "/tmp/tunnel.port";
while true; do
    sleep 300;
done

I can then connect to the server with:

function computer {
  ssh -p `cat /tmp/tunnel.port` -o NoHostAuthenticationForLocalhost=yes localhost
}

The server also has a "ClientAliveInterval 30" in /etc/ssh/sshd_config... and uses things like rbash and dss keys.

I use a random port as I have found that on occasion the port is already being used, with the intention that if I kill the connection on the server, the computer will re-connect 5 seconds later with a new port.

And because it is a random port, I need the shell script to record the port being used (not had any luck with lsof).

But the problem is that if the internet connection is lost and the tunnel closes, the computer will happily re-connect a new tunnel on a different port number, but the original process (record_port.sh) continues:

ps ax | grep "report_port"
 4166 ? Ss 0:00 /bin/bash /home/tunnel/bin/report_port.sh 2833
 6863 ? Ss 0:00 /bin/bash /home/tunnel/bin/report_port.sh 2605
20109 ? Ss 0:00 /bin/bash /home/tunnel/bin/report_port.sh 2023

pstree
init(1)─┬─...
        ├─sshd(28187)─┬─sshd(6860)───sshd(6862)───tunnel.port.sh(6863)───sleep(15533)
        │             ├─sshd... (my current shell)
        ├─tunnel.port.sh(4166)───sleep(15232)
        ├─tunnel.port.sh(20109)───sleep(15318)

Solution 1

The solution I'm using at the moment is to split the command to open the tunnel into two parts:

while true; do
    PORT=$(((RANDOM %= 1000) + 2000));
    ssh "server" "record_port.sh '$PORT';";
    ssh -N -R $PORT:localhost:22 -o ServerAliveInterval=30 "server";
    sleep 5;
done

While not perfect, it means the first ssh connection can return after specifying the port number that will be attempted, then the second ssh connection can establish the tunnel without a remote shell.

This leaves "record_port.sh" on the server simply running (with no while/sleep):

echo "$PORT" > "/tmp/tunnel.port";

Solution 2

I've excellent luck with a similar model, although without the port randomization. I had the computer I'll call "Hider" behind the remote NAT connect directly to a local, central server I'll call "Home" with a public IP. This SSH link's purpose was to create the port on Home's end used to access SSH on the Hider.

Here's the script run on Hider:

#/bin/sh
#  script: allow-ssh-from-home
unset DISPLAY
interval=30
while : ; do
    echo opening tunnel at `date +'%Y-%m-%d %H:%M:%S %Z'`
    if ssh -NC \
        -o IdentitiesOnly=yes \
        -o IdentityFile=$HOME/.ssh/id_rsa-hider-tunnel \
        -o CheckHostIP=no \
        -o StrictHostKeyChecking=no \
        -o UserKnownHostsFile=/dev/null \
        -o GSSAPIAuthentication=no \
        -o PasswordAuthentication=no \
        -o ServerAliveInterval=30 \
        -o ExitOnForwardFailure=yes \
        -R2222:127.0.0.1:22 \
        home.example.com
    then
        echo Respawning after success...
    else
        echo Delaying $interval seconds after failure... | tr -d '\012'
        sleep $interval
        echo done.  Respawning...
    fi
done
#------------------------------------eof

On Hider, the ~/.ssh/authorized_keys file has this (with some [elided]):

command="",no-user-rc,no-pty,no-X11-forwarding,no-agent-forwarding ssh-rsa AAA[more keys]RQ== rsa-hider-tunnel-key

On Home, the ~/.ssh/config has the following. Note the ServerAliveCountMax, allowing more keepalive messages to be missed (the default is 3) before the SSH on Home would otherwise die. The same config was likely present on Hider as well, although I'm not certain since it's been a while.

Host localhost
     NoHostAuthenticationForLocalhost yes

Host *
    HashKnownHosts no
    FallBackToRsh no
    ForwardAgent yes
    ForwardX11 yes
    Compression yes
    ServerAliveInterval 30
    ServerAliveCountMax 120

The usage model on Hider is to log in, then:

$ exec ssh-agent bash
$ ssh-add         # add a key known for yourself@Home
$ exec allow-ssh-from-home   # links to Home, logs out if interrupted

Now, from anywhere, log in to Home, then:

ssh -p 2222 localhost

At this point Hider would require authentication - a good idea in case someone were to break into Home.

Solution 3

autossh http://www.harding.motd.ca/autossh/ / http://packages.debian.org/squeeze/autossh can monitor the health of existing connections (by passing traffic around a local and remote port forward loop) and reconnect failed/failing ones (by re-running ssh).

netstat -tlp will give you the process id opening the remote port which might help in tracking down the new random port being used.

Comments

[Jason]

@cjc, I just want to limit it to SSH to that computer... the network that computer is on is not under my control, I don't think they will appreciate me extending their network outside of their NAT based firewall (I have explained the setup to them, and how I need to make SSH connections to the box).

[David Schwartz]

Instead, ask them how you should do it.

[Jason]

@DavidSchwartz, they have an external IT company which only works with Windows, they have no idea about SSH... Alex-north-keys below also reminded me of another advantage in that if my server was compromised, I don't want to be opening a connection back into the clients network.

[Jason]

I think autossh would be good for keeping the connection alive on the computer (the one behind the firewall)... something the first script kind of does already (I have it setup on a 5 minute cron job, where it checks if the process is already running).

[Jason]

But, I think the main problem is getting that randomly assigned port number on the server, so I can make the new connections back... netstat seems ok, but is quite difficult to parse, and wont work well with a second computer doing the same thing (I actually have 3 computers doing this, 2 at client offices, and my laptop, just in-case it goes wandering).

[Jason]

I think our two solutions are pretty similar... just that I don't want to lock it to port 2222, just in-case its in use, but also because I also happen to have 3 computers wanting to do the same kind of thing... so am wondering about making the random port number, connecting to the server to record it (ssh server ./record_port.sh $PORT)... then doing the ssh for the tunnel, but this time with -N.

[Jason]

And yes, I do like having to type in the password when connecting from the server to the remote computers (2 being on client networks, so they don't want to be effected if my server was compromised).