Boot stopped at "Started Update UTMP about System Runlevel Changes" - headless server... no X

Question

It's 8:27am. I haven't slept. I am exhausted. So I'm leaving this here and going to bed. I'll check it after. Pardon me if I'm slightly south of coherent.

Ubuntu 16.04.3 LTS server running on a VPS. Got another VPS on the same host machine, CentOS6. CentOS vps is still chugging along. Ubuntu vps won't boot. Last change was adding a Virtualmin "virtual server" (really just a separate user with privileges to some daemons). And some fiddling with postfix. Everything I've read online says to rip out my graphics drivers and reinstall them. Well I don't have any graphics drivers cause I don't have graphics. No X. No window managers. And certainly not intel or nvidia graphics drivers for X.

Closest I've come to a sane-sounding solution so far is an semi-ancient forum post about Arch Linux. Same problem caused by a missing symlink from /var/run to /run. Well, I have that symlink. So that's probably not it. And other than the tremendously unhelpful message above there is nothing to indicate what might be wrong.

Silver lining... vps has an independent VNC feature built into the hypervisor... so I've still got terminal access. And I can CTRL+ALT+F2 to get a login prompt on tty2 even with the boot messages are still stuck on tty1. No network connectivity though. A sizable chunk of the systemd services/units haven't launched. Multiple reboot have produced no meaningful changes.

It does talk about "Reached target Graphical Interface" right before everything goes tits up. But again... no X. And yes, I actually checked to make sure xorg/x11 didn't get pulled in as a dependency somehow. Screenshot below.

Solution

Turns out Ubuntu 16.04 Server comes pre-configured to boot to X even if you don't have it installed. Why? "Because cats" I suppose. Apparently, the premise, with systemd at least, is that if you don't have anything depending on the graphical.target unit (like, say, Xorg) it won't get run during boot even if it's configured as your default target. systemd "targets" multi-user.target instead in such cases. In my case, something got pulled in that depends on graphical.target causing systemd to re-order the boot sequence. This put "Started Update UTMP about System Runlevel Changes" at the end of the boot sequence. Turns out that was a red herring (sorry; I'm new to systemd and it wasn't exactly giving me much to work with).

I've changed the default target to multi-user.target (sudo systemctl set-default multi-user.target) and confirmed that this is working. It is. I've also solved the separate Shorewall issue mentioned in comments (ipsets weren't being created early enough in the boot sequence, so I made my own systemd service units and made Shorewall depend on them to correct it). Switching back to multi-user.target seems to have resolved the other services that hadn't launched (dependency tree was wrong with it launching graphical.target instead; hence, services didn't run but also didn't error out).

Part of the problem still remains: systemd is still not releasing tty1 when it reaches multi-user.target. This is hardly the end of the world as it doesn't seem to negatively affect the server's operation. But it does raise questions in my mind about what is causing it and if it might bite me on the ass later. For that reason, I'm not going to accept this answer. If someone else knows how to fix that, please, do share.

Comments

[Admin]

There is not enough log info here to debug. Something failed, but none of the logging shows anything failed. You write: "A sizable chunk of the systemd services/units haven't launched.". What do the logs say about why these units didn't launch? Were they attempted to be launched? Is there interesting output in sudo journalctl? What happens if you try to manually start the services you found not have launched during the normal boot?

[Admin]

sudo journalctl shows a few things but mostly shows success (which is obviously incorrect). The errors include jailkit not being configured, shorewall failing to init my netfilter rules, and bind9 complaining about permission denied to the rndc.key file. None of which, to my knowledge, would be related to the issue at hand. The services that are not started seem to start fine when started manually (except shorewall). Shorewall starts fine if I manualy create the ipsets it expects.

[Admin]

Given all the seemingly unrelated problems, it's likely going to be faster to move the data you need to a fresh install and start over.

[Admin]

This is a fresh install :-( . The Shorewall issue I know the cause of (and exactly how to resolve it). Jailkit simply hasn't been configured yet cause I don't even know that that is (Ubuntu 14.04 didn't have it). It's on my list. And the rndc.key file being inaccessible is a permissions error I haven't gotten around to fixing yet. All of these have known causes and fixes and would be handled in the course of configuring the vps for use... if not for the other issues described above. (more above...)

[Admin]

Scratch that, see my answer below.

[Padmanabha V]

Correct answer !! I faced this issue on Ubuntu 16.04 Lenovo laptop. I re-installed gnome-core gnome-shell. Costed arounf 220MB download. During the process it prompted me to choose the display manager lightdm or gdm3. I choose highlighted one i.e. lightdm. Post install was complete, I issued reboot command. On the first boot it entered emergency mode and prompted root password. But, next time the login screen cam up. Kudos ! I have tried rebooting 100 times to fix this and didn't try this solution only becasue it didn't have an upvote. I upvote now

[Cliff Armstrong]

It's not the correct answer for a headless server that has never had gnome installed and never will, as indicated in the original question and the title. I'm glad it's helping some people... but it's not a valid answer to the question asked.