What's the risk of upgrading over SSH?

Question

When I run

sudo do-release-upgrade

over ssh, I get the following message.

This session appears to be running under ssh. It is not recommended
to perform a upgrade over ssh currently because in case of failure it
is harder to recover.

If you continue, an additional ssh daemon will be started at port
'9004'.
Do you want to continue?

What is the real risk of upgrading over ssh? How does the additional ssh daemon help mitigate this?

Solution 1

What I would recommend doing is launching a screen session on the server and running the upgrade in screen - that way if your SSH session drops (for whatever reason) the upgrade process will not halt.

Screen is a program that allows for persistent terminal(s) on a machine. So you can start a screen session and so long as the machine is on that screen session (and it's history, running programs, etc) will continue to operate though no one user is on the machine. It was designed in the early days to provide a multi-windowed text terminal prior to the days of X Server. You can install it using APT:

sudo apt-get install screen

So you can ssh into your server, start screen, initiate the upgrade process and not have to worry about the upgrade bing botched because you've lost Internet connection or your computer crashed.

Solution 2

@Marco-Ceppi 's solution is already integrated into do-release-upgrade.

When you run do-release-upgrade it starts a screen session automatically. If your ssh session gets disconnected, you can resume the installation. All you have to do is open a new ssh session, and run do-release-upgrade again. It will reconnect to your previous installation.

A second risk, pointed out by @sepp2k is that your sshd server might need to be upgraded, and it could perhaps not restart correctly. Therefore the upgrade program runs a second deamon, at the port specified. You should check your network configuration to make sure you have access through this port, before resuming.

Good luck.

Moreover, the screen-session do-release-upgrade starts by itself is run under the root account, so if your own screen-session crashes, you will be able to recover by running sudo screen -x, if (for some reason) the command do-release-upgrade doesn't recover it by itself, which seems to be common.

Solution 3

If a new version of the ssh daemon is installed as part of the upgrade, the daemon will be restarted. If the update breaks the daemon for some reason, it would fail to start up again and you'd have no way of logging into the machine anymore.

By starting a separate sshd, which is not handled by the init system and thus won't be restarted during the upgrade, it is ensured that the old version of sshd will still be running even if the new version fails to start. Thus you can still log into the system and see what caused the daemon to break.

Solution 4

I've never (yet) had a problem doing that, though I've only upgraded half a dozen or so boxes that way. It is just that if something goes wrong the additional SSH daemon may be your only hope of avoiding a visit (or making use of whatever remote re-install options you have).

View more solutions

Comments

[baptx]

If we are using a remote server, I guess there are no real alternative to SSH, so this message looks useless. I guess using SSH is still better than doing the upgrade through the VNC recovery terminal of the web host in a web browser. But if we are using an LXC container on a local computer, an alternative to SSH is to use the command lxc-attach or lxc-console.

[Ryan C. Thompson]

Screen is an excellent tool for this situation.

[Malabarba]

I think it's more relevant that, when the daemon is restarted, your upgrade will likely be terminated unfinished, because the process is running on the ssh session.

[mgd]

Do you know from which version of Ubuntu this has been the case? I am trying to upgrade from 9.10 and after starting the upgrade (sudo do-release-upgrade) and answering "Yes", I can connect to ssh on port 9004 but there are no screen sessions listed when trying sudo screen -list . (P.S. I am not past the download stage yet.)

[mgd]

After upgrading from 9.10 to 10.04 LTS (where do-release-upgrade did not start a screen session) I am now upgrading from 10.04 LTS to 12.04 LTS and now do-release-upgrade now starts a screen session automatically.

[Juh_]

So basically, do-release-upgrade take care of everything before hand, then: 1) you can reconnect from a broken ssh by re-running do-release-upgrade after re-logging and 2) you can re-log from a failed sshd upgrade through the port specified at the beginning by do-release-upgrade. Is that it ?

[manu]

It seems that do-release-upgrade (since precise) is automatically executing screen.

[Felipe]

sudo screen -x saved my life! Thanks.

[Gradyn Wursten]

screen is no longer avalible, and this does not seem to work with TMUX for some reason. It returns this error: === Command terminated with exit status 1 (Wed Oct 19 21:14:13 2016) ===

[Jahmic]

Then why is the warning message still there? Sounds very ominous.

[Peter]

Make sure screen is installed as it can only be used if it is installed. You might have to install it with apt-get install screen. At least this is what I realized when upgrading from Ubuntu 17.10 to 18.04. screen was not installed and not used on my first attempt. I stopped the upgrade before the download, installed screen, and run do-release-upgrade again. Then screen was used.

[Ryan Allen]

screen -dmS do-release-upgrade -> screen -r to switch to the running screen -> Ctrl + A, Ctrl + D to leave the screen running in the background and go back to the main screen