Updated 16.04 server to 18.04.1, now I see weird errors about missing files, failing to resolve dns, failing to connect to bus

Question

Ok, so this is an upgrade from 16.04.1 LTS to 18.04.1 LTS, the server is headless. After the upgrade is finished, and a reboot is completed, the following happens:

Trying to upgrade packages only results in the following:

~$ sudo apt upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following packages will be upgraded:
  policykit-1 screen smartmontools
3 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
1 not fully installed or removed.
Need to get 0 B/1,081 kB of archives.
After this operation, 147 kB of additional disk space will be used.
Do you want to continue? [Y/n] y

(Reading database ... 159133 files and directories currently installed.)
Preparing to unpack .../smartmontools_6.5+svn4324-1_amd64.deb ...
Failed to connect to bus: No such file or directory
[...]

Is it Dbus it is talking about? Anyway, how about a restore?

~$ sudo apt update
Err:1 http://security.ubuntu.com/ubuntu bionic-security InRelease
  Temporary failure resolving 'security.ubuntu.com'
Err:2 http://se.archive.ubuntu.com/ubuntu bionic InRelease
  Temporary failure resolving 'se.archive.ubuntu.com'
Err:3 http://se.archive.ubuntu.com/ubuntu bionic-updates InRelease
  Temporary failure resolving 'se.archive.ubuntu.com'
Err:4 http://se.archive.ubuntu.com/ubuntu bionic-backports InRelease
  Temporary failure resolving 'se.archive.ubuntu.com'
Reading package lists... Done
Building dependency tree
Reading state information... Done
1 package can be upgraded. Run 'apt list --upgradable' to see it.
W: Failed to fetch http://se.archive.ubuntu.com/ubuntu/dists/bionic/InRelease  Temporary failure resolving 'se.archive.ubuntu.com'
W: Failed to fetch http://se.archive.ubuntu.com/ubuntu/dists/bionic-updates/InRelease  Temporary failure resolving 'se.archive.ubuntu.com'
W: Failed to fetch http://se.archive.ubuntu.com/ubuntu/dists/bionic-backports/InRelease  Temporary failure resolving 'se.archive.ubuntu.com'
W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/bionic-security/InRelease  Temporary failure resolving 'security.ubuntu.com'
W: Some index files failed to download. They have been ignored, or old ones used instead.

What? Is dns broken?

~$ wget www.google.com
--2018-11-10 01:55:16--  http://www.google.com/
Resolving www.google.com (www.google.com)... failed: Temporary failure in name resolution.
wget: unable to resolve host address ‘www.google.com’

Looks like it. Hmm, ifconfig doesn't show dns. Google directs me to this question , let's try it.

~$ systemd-resolve --status
sd_bus_open_system: No such file or directory

Ok, I'm more and more lost now. Google now finds this question, which is about docker, which is not my case, but also mentions a problem where systemd isnt pid 1:

~$ ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.1  0.1 159752  9116 ?        Ss   01:25   0:02 /sbin/init splash nomdmonddf nomdmonisw
root         2  0.0  0.0      0     0 ?        S    01:25   0:00 [kthreadd]
root         3  0.0  0.0      0     0 ?        S    01:25   0:00 [ksoftirqd/0]
root         5  0.0  0.0      0     0 ?        S<   01:25   0:00 [kworker/0:0H]
root         7  0.0  0.0      0     0 ?        S    01:25   0:00 [rcu_sched]
root         8  0.0  0.0      0     0 ?        S    01:25   0:00 [rcu_bh]
root         9  0.0  0.0      0     0 ?        S    01:25   0:00 [migration/0]
root        10  0.0  0.0      0     0 ?        S    01:25   0:00 [watchdog/0]
root        11  0.0  0.0      0     0 ?        S    01:25   0:00 [watchdog/1]
[...]

I guess? Is this the root problem, or a symptom? How can I fix this?

Solution

I had the same error on one OVH VPS server after upgrading to 18.04.01. I have other very similar servers working fine after the upgrade, which helped me fix this.

Went down a rabbit hole and noticed:

sudo systemctl
...
systemd-logind.service     loaded failed failed
...

Which lead me to this site: https://forum.proxmox.com/threads/systemd-logind-failures.44219/

I confirmed that my /var/run was not sym linked to /run , and it was on the other servers. This one had files in both places.

I symlinkd almost as suggested, fixing "directory not empty" errors in every line with:

mv -f /var/run/sudo/ts/* /run/sudo/ts/; rm -rf /var/run/sudo/ts; 
mv -f /var/run/sudo/* /run/sudo/; rm -rf /var/run/sudo; 
mv -f /var/run/* /run/; rm -rf /var/run; 
ln -s /run /var/run;
reboot

This step solved parts of the problem, such as:

$ systemd-resolve --status

But the DNS resolution was still failing, but working fine using resolve:

$ ping google.com
ping: google.com: Temporary failure in name resolution

$ systemd-resolve google.com
google.com: 216.58.213.174

As per the suggestion in https://superuser.com/questions/1317623/nslookup-failed-but-systemd-resolved-works

I changed my /etc/resolv.conf symlink from /run/resolvconf/resolv.conf to /run/systemd/resolve/resolv.conf

$ ls -l /etc/resolv.conf
lrwxrwxrwx 1 root root 29 Feb 24  2017 /etc/resolv.conf -> ../run/resolvconf/resolv.conf

$ sudo rm /etc/resolv.conf
$ sudo ln -s /run/systemd/resolve/resolv.conf /etc/resolv.conf

$ ping google.com
PING google.com (216.58.204.238) 56(84) bytes of data.
64 bytes from par21s06-in-f14.1e100.net (216.58.204.238): icmp_seq=1 ttl=53 time=9.11 ms

Everything is working now.