Failed to activate service 'org.freedesktop.systemd1': timed out

Question

On ubuntu 16.04.6 on trying to mask a systemd service. I am seeing errors like below.

systemctl mask hadoop-hdfs-zkfc.service
Failed to activate service 'org.freedesktop.systemd1': timed out

I am using the default systemd version that comes with Ubuntu 16.04.6.

ubuntu@platform1:~$ systemctl --version
systemd 229
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP
+LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 +SECCOMP +BLKID +ELFUTILS
+KMOD -IDN

The issue does not come always. But once this issue is hit the only way to recover the systemd is to do a hard-reboot.

Looking at the syslog it appears systemd aborted.

May 18 08:49:24 platform3 systemd[1]: Removed slice User Slice of support.
May 18 08:49:27 platform3 systemd[1]: Assertion 's->type ==
SERVICE_ONESHOT' failed at ../src/core/service.c:1792, function
service_enter_start(). Aborting.
May 18 08:49:27 platform3 systemd[1]: Caught <ABRT>, dumped core as pid 15839.
May 18 08:49:27 platform3 systemd[1]: Freezing execution. 

busctl output looks like below

ubuntu@platform3:~/logs$ busctl
NAME                               PID PROCESS         USER             CONNECTION    UNIT                      SESSION    DESCRIPTION
:1.1                               976 systemd-logind  root             :1.1          systemd-logind.service    -          -
:1.3                               971 accounts-daemon root             :1.3          accounts-daemon.service   -          -
:1.5434                          49174 systemctl       root             :1.5434       cron.service              -          -
:1.5435                          49223 systemctl       root             :1.5435       ssh.service               -          -
:1.5436                          49408 busctl          ubuntu           :1.5436       ssh.service               -          -
:1.7                              1109 unattended-upgr root             :1.7          unattended-upgrades.se... -          -
com.ubuntu.LanguageSelector          - -               -                (activatable) -                         -
org.debian.AptXapianIndex            - -               -                (activatable) -                         -
org.freedesktop.Accounts           971 accounts-daemon root             :1.3          accounts-daemon.service   -          -
org.freedesktop.DBus               936 dbus-daemon     messagebus       org.freedesktop.DBus dbus.service              -          -
org.freedesktop.hostname1            - -               -                (activatable) -                         -
org.freedesktop.locale1              - -               -                (activatable) -                         -
org.freedesktop.login1             976 systemd-logind  root             :1.1          systemd-logind.service    -          -
org.freedesktop.network1             - -               -                (activatable) -                         -
org.freedesktop.resolve1             - -               -                (activatable) -                         -
org.freedesktop.systemd1             - -               -                (activatable) -                         -
org.freedesktop.thermald             - -               -                (activatable) -                         -
org.freedesktop.timedate1            - -               -                (activatable) -                         - 

Can someone let me know, how to debug this?

The issue looks similar to the one discussed here. So is this some known systemd issue on Ubuntu 16.04.6?

Solution 1

This has been answered in systemd mailing list. Cross-posting the answer

This bug (https://github.com/systemd/systemd/issues/4444) was fixed in systemd v236.

See if you can use a newer version of systemd.

Solution 2

As it clear from looking through the output of the busctl call published above by you, your org.freedesktop.systemd1 daemon (init.scope unit) isn't active actually:

org.freedesktop.systemd1             - -               -                (activatable) -                         -

However it might be activated like this:

$ systemctl daemon-reexec

Or this:

$ sudo kill 1

Which should ask systemd, in a little bit more insistent manner - by sending a SIGTERM signal to systemd, to do essentially the same - run daemon-reexec. I encountred a look-alike problem after depleting all the free RAM while not having a swap file on the system. And reexecing the systemd daemon totally solved it for me without the need to reboot the machine (Though before it you would need to kill some user processes to free some RAM if the problem was caused in the same way as in my case). As it stated in the systemd's man pages, daemon-reexec is save to use:

   daemon-reexec
       Reexecute the systemd manager. This will serialize the manager
       state, reexecute the process and deserialize the state again. This
       command is of little use except for debugging and package upgrades.
       Sometimes, it might be helpful as a heavy-weight daemon-reload.
       While the daemon is being reexecuted, all sockets systemd listening
       on behalf of user configuration will stay accessible.

Comments

[Moberg]

I have this problem regularly since last 2 weeks during heavhy load (I think). Is it possible to diagnose if it is a problem of depleting RAM? Usually it takes more than 3 hour to recover.

[Ale]

Only solution that worked for me on Ubuntu 18.