SSH failed to start - Missing privilege separation directory: /var/run/sshd

Question

I have a VPS running Ubuntu 16.04.5 that's been going for a number of years now with little issue. Today, however, I found I was unable to access the server using SSH, receiving 'connection refused' errors. I accessed the server using my VPS host's serial console service, and traced the issue down to openssh server failing to start. Here's an output of service status, service start, and sshd -t following a fresh reboot:

root@167:/# service ssh status
● ssh.service - OpenBSD Secure Shell server
   Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
   Active: failed (Result: start-limit-hit) since Fri 2019-01-18 04:56:42 EST; 24min ago
  Process: 983 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=255)

Jan 18 04:56:42 167 systemd[1]: Failed to start OpenBSD Secure Shell server.
Jan 18 04:56:42 167 systemd[1]: ssh.service: Unit entered failed state.
Jan 18 04:56:42 167 systemd[1]: ssh.service: Failed with result 'exit-code'.
Jan 18 04:56:42 167 systemd[1]: ssh.service: Service hold-off time over, scheduling restart.
Jan 18 04:56:42 167 systemd[1]: Stopped OpenBSD Secure Shell server.
Jan 18 04:56:42 167 systemd[1]: ssh.service: Start request repeated too quickly.
Jan 18 04:56:42 167 systemd[1]: Failed to start OpenBSD Secure Shell server.
Jan 18 04:56:42 167 systemd[1]: ssh.service: Unit entered failed state.
Jan 18 04:56:42 167 systemd[1]: ssh.service: Failed with result 'start-limit-hit'.
root@167:/# service ssh start
Job for ssh.service failed because the control process exited with error code. See "systemctl status ssh.service" and "journalctl -xe" for details.
root@167:/# sshd -t
Missing privilege separation directory: /var/run/sshd

I've attempted some research into this, but nothing that's come up seems to have an actual solution - Just endless cycles of 'I have this problem' with no answers, answers that are outdated, or just generally unhelpful information.

Does anybody have any ideas on what to do next to troubleshoot/resolve this issue? SSH was last working about 12 hours ago when I logged in to run updates and rebooted the server.

UPDATE

Issue was resolved using workaround 1 as provided here: https://askubuntu.com/a/1110843/531533

Solution

According to the bug report here, the problem can be worked around by adding the directory creation code to /etc/rc.local.

I've done some research and I still can't figure out what is supposed to create that directory on a normal server startup - there're a couple of scripts under /etc that do this, but they appear to be deprecated as they relate to the old SysV startup and I can't find anything in the SystemD init setup to create that directory.

[Update]: Just noticed @pa4080 comment, and this seems to be it - the file /usr/lib/tmpfiles.d/sshd.conf exists and contains the line d /var/run/sshd 0755 root root, in openssh-server version 1:7.2p2-4ubuntu2.6.

The tmpfiles.d directory is part of the systemd-tmpfiles service and is documented here .

The could be an update anomaly as I've seen relevant bug reports that discuss this as a failed update, and updating the openssh-server to the latest version might fix it.

Comments

[SierraKomodo]

The version for openssh matches what you're showing - It doesn't look like updating is an option right now, as apt states this is the newest version. I'll see if a patch comes out later today. If not, I'll give @pa4080's suggestion a try - I'm not too keen on making changes to a system file that was suggested back in '08, so I probably won't be trying the update to rc.local.

[Guss]

Check the file specified by pa4080 - does it have the a line like in my updated answer?

[SierraKomodo]

It does - Exactly as described: d /var/run/sshd 0755 root root

[Guss]

On a 16.04 test system I got, running systemd-tmpfiles --create causes the directory to be created. This command should have been run by the SystemD init process on boot, but if there was no restart after an update that changed the behavior for how openssh-server uses that directory, that could explain the problem you had.

[pa4080]

SierraKomodo, I just post an answer where I've described also a third method by using crontab.

[SierraKomodo]

Modifying /usr/lib/tmpfiles.d/sshd.conf resolved the issue.