apt/unattended-upgrades stalls shutdown

Question

When unattended-upgrades is installed, 9 out of 10 shutdowns/reboots hang while "starting unattended upgrades shutdown". This hang stalls the shutdown process for 5-10 mins.

If I disable unnattended-upgrades via the /etc/apt/apt.conf.d/20auto-upgrades and/or 50unattended-upgrades, the problems occurs.

If I terminate the service before shutdown/reboot (sudo service unattended-upgrades stop) the problem still occurs.

If I remove the package (sudo apt remove unattended-upgrades) the problem no longer occurs.

This occurs on a freshly installed version of Ubuntu Server 16.04.1 (both unattended-upgrades installed via install GUI or manual install of unattended-upgrades)

Both Kern.log & syslog do not show the shutdown process (I believe because the filesystems have already unmounted)

Has anyone else seen or fixed this issue? Going crazy trying to troubleshoot it.

Solution 1

Looking around to get closer to the root cause

The problem seems to be the script running at shutdown.

I identified the corresponding file with:

find /etc/systemd -name *unattended*

which gaves me the related systemd script:

/etc/systemd/system/shutdown.target.wants/unattended-upgrades.service

which then told me the script executed on shutdown:

/usr/share/unattended-upgrades/unattended-upgrade-shutdown

Investigating deeper to find the root cause

within this script there is a section in line 120 related to the section in /etc/apt/apt.conf.d/50unattended-upgrades -> Unattended-Upgrade::InstallOnShutdown

Line 120 of /usr/share/unattended-upgrades/unattended-upgrade-shutdown:

if apt_pkg.config.find_b("Unattended-Upgrade::InstallOnShutdown", False):

The problem: it expects the keyword "False" while in the apt conf we should add "false" (exact string comparison)!

Solution

I was able to fix/workaround the stalling shutdown in 3 different ways:

Workaround A

• write "False" instead of "false" in /etc/apt/apt.conf.d/50unattended-upgrades

This setting is upgrade safe until a real fix is provided because the file we change here gets not overwritten by an update of unattended-upgrades. Problem: When the root cause gets fixed this will result in a stalling shutdown again so I suggest to combine this with Workaround B.

OR: Workaround B

• decrease the wait time in /etc/systemd/system/shutdown.target.wants/unattended-upgrades.service from default to 15 seconds:

vim /etc/systemd/system/shutdown.target.wants/unattended-upgrades.service

[Service]
Type=oneshot
ExecStart=/usr/share/unattended-upgrades/unattended-upgrade-shutdown
TimeoutStartSec=15

This setting is NOT upgrade safe because the file we change here may get overwritten by an update of unattended-upgrades. Besides this it is really far away from fixing something but it will ensure that your system will not wait several minutes when shutting down. Keep in mind that after an upgrade of unattended-upgrades you may have to set this again!

OR: Fix C (have to be reported upstream)

• fix /usr/share/unattended-upgrades/unattended-upgrades-shutdown to expect "false" instead of "False"

patching /usr/share/unattended-upgrades/unattended-upgrade-shutdown:

--- /tmp/unattended-upgrade-shutdown    2017-02-03 14:53:03.238103238 +0100
+++ /tmp/unattended-upgrade-shutdown_fix    2017-02-03 14:53:17.685589001 +0100
@@ -117,7 +117,7 @@
     # run it
     p = None
     apt_pkg.init_config()
-    if apt_pkg.config.find_b("Unattended-Upgrade::InstallOnShutdown", False):
+    if apt_pkg.config.find_b("Unattended-Upgrade::InstallOnShutdown", false):
         env = copy.copy(os.environ)
         env["UNATTENDED_UPGRADES_FORCE_INSTALL_ON_SHUTDOWN"] = "1"
         logging.debug("starting unattended-upgrades in shutdown mode")

Conclusion

tbh only the last one is a real fix. the both other options are just workarounds until the real fix would be implemented.

This has to be done upstream and as this affects both Debian (tested on Debian Stretch) and Ubuntu (tested on Ubuntu 16.04.1) for both distributions.

I have opened a bug report here: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1661611

Solution 2

This has been fixed in systemd

https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600

(original report: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1661611)

Comments

[user535733]

Unable to reproduce in a 16.04.1 test VM. Shutdown is not delayed here.

[garullon245136]

Could it be hardware based? I'm not entirely sure what happens with unattended-upgrades during shutdown.

[user535733]

I'm wondering why u-u is always running at the time of shutdown: u-u is not a daemon; it's merely a script that runs briefly once each day and then terminates.

[garullon245136]

It seems as though the shutdown process tries to run u-u during the stage where all of the filesystems are being dismounted.This doesn't seem to be controlled by the /etc/rc6.d/ or /etc/rc0.d/ links as I have removed all of the links and the process still runs during shutdown.

[user535733]

16.04 uses systemd, not sysvinit. The /etc/rc* directories still exist for sysvinit compatibility, they are no longer the primary way of determining boot/shutdown tasks.

[user535733]

Look in /etc/apt/apt/conf.d/50unattended-upgrades for the 'run u-u at shutdown' option (around line 25). Ensure it is 'false' or commented out.

[Brian Murray]

apt_pkg.config.find_b() returns a boolean not a string. "find_b(key[, default=False]) → bool Return the boolean value stored at key, or the value given by the bool object default if the requested option is not set." apt.alioth.debian.org/python-apt-doc/library/… So it's not a bug in unattended-upgrade-shutdown as checking for False is correct.

[Admin]

As anyone who follows the above Launchpad bug report link will see, this issue was not in fact caused by a failing string comparison, but instead was due to a systemd sequencing bug that has reportedly now been fixed.