How to reliably keep an SSH tunnel open?

Question

I use an SSH tunnel from work to go around various idotic firewalls (it's ok with my boss :)). The problem is, after a while the ssh connection usually hangs, and the tunnel is broken.

If I could at least monitor the tunnel automatically, I could restart the tunnel when it hangs, but I haven't even figured a way of doing that.

Bonus points for the one who can tell me how to prevent my ssh connection from hanging, of course!

Solution 1

Sounds like you need autossh. This will monitor an ssh tunnel and restart it as needed. We've used it for a couple of years and it seems to work well.

autossh -M 20000 -f -N your_public_server -R 1234:localhost:22 -C

More details on the -M parameter here

Solution 2

All stateful firewalls forget about a connection after not seeing a packet for that connection for some time (to prevent the state tables from becoming full of connections where both ends died without closing the connection). Most TCP implementations will send a keepalive packet after a long time without hearing from the other side (2 hours is a common value). If, however, there is a stateful firewall which forgets about the connection before the keepalive packets can be sent, a long-lived but idle connection will die.

If that is the case, the solution is to prevent the connection from becoming idle. OpenSSH has an option called ServerAliveInterval which can be used to prevent the connection from being idle for too long (as a bonus, it will detect when the peer died sooner even if the connection is idle).

Solution 3

I've used the following Bash script to keep spawning new ssh tunnels when the previous one dies. Using a script is handy when you don't want or can't install additional packages or use compiler.

while true
do
  ssh <ssh_options> [user@]hostname
  sleep 15
done

Note that this requires a keyfile to establish the connection automatically but that is the case with autossh, too.

Solution 4

Systemd is ideally suited for this.

Create a service file /etc/systemd/system/sshtunnel.service containing:

[Unit]
Description=SSH Tunnel
After=network.target

[Service]
Restart=always
RestartSec=20
User=sshtunnel
ExecStart=/bin/ssh -NT -o ServerAliveInterval=60 -L 5900:localhost:5900 user@otherserver

[Install]
WantedBy=multi-user.target

(Modify the ssh command to suit)

• this will run as user sshtunnel so make sure that user exists first
• issue systemctl enable sshtunnel to set it to start at boot time
• issue systemctl start sshtunnel to start immediately

---

Update Jan 2018: some distros (e.g. Fedora 27) may use SELinux policy to prevent the use of SSH from systemd init, in which case a custom policy will need to be created to provide the necessary exemptions.

Solution 5

On your own mac or linux machine configure your ssh keep the server ssh alive every 3 minutes. Open a terminal and go your your invisible .ssh in your home:

cd ~/.ssh/ 

then create a 1 line config file with:

echo "ServerAliveInterval 180" >> config

you should also add:

ServerAliveCountMax xxxx (high number)

the default is 3 so ServerAliveInterval 180 will stop sending after 9 minutes (3 of the 3-minute interval specified by ServerAliveInterval).

View more solutions

Comments

[m3nda]

It is your tunnel dead because inactivity? I had this problem when tunneling ports from my phone so i finally ended spawning dummy commands on the connection to make it "alive" using the watch command like: watch -n1 60 echo "wiiiii". Tunnel will not die unless network is broken or you don't use it.

[Admin]

Related: unix.stackexchange.com/q/200239

[akent]

+1 for autossh, it does what it says on the tin. I believe part of its functionality also is to send keep-alive style packets to prevent any kind of timeout.

[akent]

... but the question was about how to keep SSH tunnels open, not just a terminal session. Screen IS great though!

[Darren Hall]

The interval specified is in seconds, so you can provide some fine tuning. If your stateful firewall has a 5 minute idle timeout, then 60 or 120 seconds is enough to keep the connection open. It's one of the ways I keep my ssh sessions through my home router open.

[Peltier]

Note that your command is not recommended if you already have a config file. Using >> for redirection would be a lot better!

[JWL]

why does ServerAliveInterval 180 give us 6 minutes? intuition makes me try this : 180/60 == 3. So, does ServerAliveInterval work in multiples of 30 secs?

[metamatt]

@mcnemesis: ServerAliveInterval 180 means 3 minutes. ServerAliveCountMax default of 3 means 3 of those intervals, so 9 minutes.

[metamatt]

I'm voting this answer up because thanks for mentioning ServerAliveCountMax, and what happens if you specify ServerAliveInterval without ServerAliveCountMax. But like the preceding comments, I notice the calculation on "will stop sending after" is wrong, and I think this answer would serve better if it just gave the information on these options, not telling us how to apply them with cd and echo commands.

[metamatt]

Thanks, this helped. But note (from a lower-ranked answer here, superuser.com/a/146641/115515) that if you specify ServerAliveInterval and not ServerAliveCountMax, you may find ssh intentionally disconnecting sooner than you wanted.

[Jimit H.]

You should add any reasons that you'd use this script over autossh, or is it just that it's easier this way?

[nafg]

This wouldn't help if ssh itself freezes, would it?

[Lambart]

Every 15 seconds seems pretty often to ping the server.

[Lambart]

Downvoting because it makes no sense to set ServerAliveCountMax to a "high number". ServerAliveCountMax specifies how many times it will try to send the "keepalive" message before giving up. The default is 3, so with ServerAliveInterval 180, it will stop sending ONLY if the server has NOT RESPONDED after 9 minutes, in which case your connection is probably well and truly dead.

[Lambart]

@metamatt, that lower-ranked answer you reference is lower-ranked for good reason: IT IS WRONG.

[user1146334]

+1 for inline option in case you don't want it for all of your SSH connections

[binki]

@Lambart but if the connection is really flaky and drops connections often, it at least detects a dead connection and gives the opportunity to retry earlier.

[juckele]

autossh -f -nNT -i ~/keypair.pem -R 2000:localhost:22 [email protected] You might notice that I set this up using -nNT which doesn't create a remote terminal so that I can put autossh into the background, and the -i option for SSH to use a .pem file. If you're going to be keeping a connection open all the time, I definitely recommend going through the extra setup.

[rinogo]

For what it's worth, it looks like it's typically better to omit the -M parameter: bugs.debian.org/cgi-bin/bugreport.cgi?bug=351162

[Mads Skjern]

You write "in case connectivity goes down entirely". Now I don't understand, what problems does autossh fix itself, and what does it not? I thought, of course, it would take care of any broken connection, like unplugging the cable for a few hours, but perhaps not?

[muttonUp]

If its not working and you are using keys, check this answer - serverfault.com/a/545093/288788

[PedroD]

How can I make autossh retry even if the first connection ever fails?

[quarkex]

It helps if you can't install things in the server. autossh doesn't come preinstalled and bureucracy it's sometimes very obtuse.

[Raúl Salinas-Monteagudo]

This is a very good idea. Even autossh is useless if the previous connection is perceived as timed out earlier in the remote side than in the local host, since in this case the local host will try to connect again, but the forwarding cannot be established because the port is still open.

[Luke Stanley]

I did this to make it retry upon network change, it works well for me: autossh -M 0 -o "ServerAliveInterval 10" -o "ServerAliveCountMax 2" -L 9999:localhost:19999 [email protected]

[Yannick Y]

How does that work if I have a passphrase ? would it store that passphrase in a keychain ( for example on a mac ) ?

[guettli]

This looks very similar to my gist: gist.github.com/guettli/… Feedback is welcome!

[David Tonhofer]

Excellent for a systemd system. If one uses Restart=on-failure then manually killing the SSH client will not result in a restart-by-systemd as the SSH client with exit with success.

[David Tonhofer]

If you want to start ssh from a (bash) script given as argument to ExecStart for example to build the ssh argument list, do basic checks etc then call it from the script like so exec /bin/ssh -N .... Here is my command: exec /bin/ssh -N -oExitOnForwardFailure=Yes -oTCPKeepAlive=no -oServerAliveInterval=5 -oServerAliveCountMax=6 -i "${LOCAL_PRIVATE_KEY}" -L "${TUNNEL_INLET}:${TUNNEL_OUTLET}" "${REMOTE_USER}@${REMOTE_MACHINE}" where TUNNEL_INLET="127.0.0.1:3307" and TUNNEL_OUTLET="127.0.0.1:3306"

[sudo]

Yes, preferable not to have to install things. I've been doing it this way for a year as my only way to keep a remote machine accessible (even set crontab to run it upon reboot). It's never failed, and more importantly, I know why it will never fail.

[cmal]

Just used this. But after 10 minutes, the ssh connection is dead, even though I had added a 20 secs AUTOSSH_POLL interval.

[Jack Wasey]

mosh is also a great tool, but you need to install on server.

[confetti]

For those who just copy and then wonder why their connection might be slow: -C forces compression of the connection, it's useful on modem lines and slow internet connections but if a fast connection is in place it will actually slow things down. Nowadays it should most likely be not used, unless your connection speed is terribly slow.

[4F2E4A2E]

Not working on macOS 10.15.3

[4F2E4A2E]

Not working on macOS 10.15.3

[Tommy Bravo]

@rinogo bugs.debian.org/cgi-bin/bugreport.cgi?bug=351162#34 only if you're using version 1.4a-1 or above in Debian.

[Brian C]

DO NOT increase ServerAliveCountMax to a huge number - you are effectively disabling keepalive by doing so. ServerAliveCountMax limits the number of missed replies, not the number of successes, and terminates the connection if the other side is not responding. The understanding that it limits the overall amount of "keepalive" packets sent is based on a misreading of the manual. Thirty seconds of Googling will make this very clear.

[Hayden Thring]

This is amazingly simple and very tempting :) an alternative though not as secure to key auth is sshpass command.

[Yeti]

You forgot about adding StartLimitIntervalSec=0 to the [Unit] section. This disables the rate-limiting feature of systemd, which prevents restarting services if they fail too fast. This could happen if the local network device is temporarily down, and ssh would quit immediately with a connection refused error.

[Yeti]

You forgot about adding StartLimitIntervalSec=0 to the [Unit] section. This disables the rate-limiting feature of systemd, which prevents restarting services if they fail too fast. This could happen if the local network device is temporarily down, and ssh would quit immediately with a connection refused error.

[CPBL]

Fails for me. ie the tunnel isn't open. Where do I find the error messages from systemd? When I try to use the tunnel, I get: channel 0: open failed: connect failed: Connection refused stdio forwarding failed kex_exchange_identification: Connection closed by remote host Connection closed by UNKNOWN port 65535 but 65535 was not mentioned in my command. My tunnel works fine if I set it up myself.

[Admin]

i tried this out and thought it was great but just noticed the network speed was getting nightmarishly slow