Apache2 fails to start on boot with Ubuntu 16.04

Question

I have just created a new Ubuntu 16.04 server (on digital ocean, if it matters). I have set up Apache2, and it runs correctly, but on every restart it fails to boot.

I get the following from systemctl status:

root@twl-ubuntu-2gb-sgp1-01:~# systemctl status apache2.service 
● apache2.service - LSB: Apache2 web server
   Loaded: loaded (/etc/init.d/apache2; static; vendor preset: enabled)
  Drop-In: /lib/systemd/system/apache2.service.d
           └─apache2-systemd.conf
   Active: inactive (dead)
     Docs: man:systemd-sysv-generator(8)

If I run systemctl is-enabled I get "static":

root@twl-ubuntu-2gb-sgp1-01:~# systemctl is-enabled apache2.service 
static

Even after running systemctl enable it doesn't change:

root@twl-ubuntu-2gb-sgp1-01:~# systemctl enable apache2.service 
Synchronizing state of apache2.service with SysV init with /lib/systemd/systemd-sysv-install...
Executing /lib/systemd/systemd-sysv-install enable apache2
root@twl-ubuntu-2gb-sgp1-01:~# systemctl is-enabled apache2.service 
static

But apache has no problems starting. When I run systemctl start manually it works just fine:

root@twl-ubuntu-2gb-sgp1-01:~# systemctl start apache2.service 
root@twl-ubuntu-2gb-sgp1-01:~# systemctl status apache2.service 
● apache2.service - LSB: Apache2 web server
   Loaded: loaded (/etc/init.d/apache2; static; vendor preset: enabled)
  Drop-In: /lib/systemd/system/apache2.service.d
           └─apache2-systemd.conf
   Active: active (running) since Sat 2016-07-09 07:33:30 EDT; 8s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 1847 ExecStart=/etc/init.d/apache2 start (code=exited, status=0/SUCCESS)
    Tasks: 7
   Memory: 22.5M
      CPU: 455ms
   CGroup: /system.slice/apache2.service
           ├─1871 /usr/sbin/apache2 -k start
           ├─1875 /usr/sbin/apache2 -k start
           ├─1876 /usr/sbin/apache2 -k start
           ├─1877 /usr/sbin/apache2 -k start
           ├─1878 /usr/sbin/apache2 -k start
           └─1879 /usr/sbin/apache2 -k start

Jul 09 07:33:28 twl-ubuntu-2gb-sgp1-01 systemd[1]: Starting LSB: Apache2 web server...
Jul 09 07:33:28 twl-ubuntu-2gb-sgp1-01 apache2[1847]:  * Starting Apache httpd web server apache2
Jul 09 07:33:29 twl-ubuntu-2gb-sgp1-01 apache2[1847]: AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set t
Jul 09 07:33:30 twl-ubuntu-2gb-sgp1-01 apache2[1847]:  *
Jul 09 07:33:30 twl-ubuntu-2gb-sgp1-01 systemd[1]: Started LSB: Apache2 web server.

I'm new to systemd, only recently upgraded to 16.04 from 14.04. Where am I going wrong, or perhaps, what can I do to debug why apache2 is refusing to start?

Update 1 (2016-07-14)

It turns out Apache is being loaded, but is receiving a SIGTERM as soon as it launches. The only line in the log file is:

[Wed Jul 13 21:37:15.730331 2016] [mpm_prefork:notice] [pid 1871] AH00169: caught SIGTERM, shutting down

Any ideas how to get a better error message? As I mentioned running systemctl start apache2 immediately after boot starts everything just fine.

Update 2 (2016-07-15)

It turns out that Sigterm was from it being killed when I ran sudo reboot. There were no log messages in /var/log/apache2/error.log. If I run journalctl I get these lines during the bootup process, so it seems to work correctly, or at least, no errors are logged:

Jul 13 21:38:04 twl-ubuntu-2gb-sgp1-01 systemd[1]: Starting LSB: Apache2 web server...
Jul 13 21:38:04 twl-ubuntu-2gb-sgp1-01 apache2[1800]:  * Starting Apache httpd web server apache2
Jul 13 21:38:04 twl-ubuntu-2gb-sgp1-01 apache2[1800]: AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set t
Jul 13 21:38:05 twl-ubuntu-2gb-sgp1-01 apache2[1800]:  *
Jul 13 21:38:05 twl-ubuntu-2gb-sgp1-01 systemd[1]: Started LSB: Apache2 web server.

But then the service is never accessible, and systemctl status apache2.service still reports it as "inactive (dead)"

Solution

I looked up what the the static state of is-enabled means, and found that:

In this context, static means that the unit file does not contain an "install" section, which is used to enable a unit. As such, these units cannot be enabled.

Also, I found that the Apache2 package contains only a systemd include file, not a complete file:

 # /lib/systemd/system/apache2.service.d/apache2-systemd.conf
 [Service]
 Type=forking
 RemainAfterExit=no

It appears that the remainder of the logic for managing the Apache2 process comes from using a shim that allows systemd to run SysVinit scripts. So try enabling Apache2 to run at boot using a symlink in a SysVinit directory:

ln -s /etc/init.d/apache2 /etc/rc5.d/S02apache2

It seems like a bug or at least a missing feature that Apache2 isn't shipping with a full systemd 'service' file on Ubuntu 16.04.

Comments

[Jos]

Anything useful in /var/log/apache2/error.log?

[Jason O'Neil]

@Jos Thanks for the hint. After a fresh boot I find this: [Wed Jul 13 21:37:15.730331 2016] [mpm_prefork:notice] [pid 1871] AH00169: caught SIGTERM, shutting down That's the only line in the log between boot and when I restart the service manually. Any ideas on how to get a richer error message?

[Mark Stosberg]

try grep -R SIGTERM /var/log/*. Maybe the sender of the SIGTERM has also logged why the SIGTERM is being sent. A key word in the the logging is caught. Apache isn't killing itself. Something else is.

[Jason O'Neil]

Thanks for the hint @MarkStosberg. Sadly it turns out that SIGTERM was just being logged from the "sudo reboot" I had just run. When I look through the logs in journalctl it logs that apache2 was started without issue. I'm at a loss for what to try next... perhaps just a hackish cronjob to restart the service a few seconds after boot?

[Mark Stosberg]

For what to try next, I would consider Nginx if it's a viable option for you. I managed Apache servers for over a decade and have found Nginx easy to learn and pleasant to use for a variety of needs. I've also tested it on Ubuntu 16.04 with systemd. The Apache packages for Ubuntu 16.04 could use love. Fedora has a pure-systemd Apache solution, so there's no reason that Ubuntu to be using this hack for Apache management that's half systemd and half old SysV init scripts. I suspect that complexity is at the root of your problem.

[Mark Stosberg]

BTW, there are lots of hits for this kind of SIGTERM case if you want to look through them, including a few other hits in the StackExchange network: duckduckgo.com/?q=mpm_prefork+caught+SIGTERM

[Jason O'Neil]

Thanks for the tips @MarkStosberg, I'll see if I can resolve the problem but if not I'll go for the nginx transition. I've used it on other servers and do find it really nice, it's only been the issue of adapting the config that held me back. But given the time I'm sinking into getting apache to work anyway now might be the time to make the switch. Cheers

[Jos]

I have one on my Ubuntu Server 16.04 though.

[Mark Stosberg]

See updated answer.

[Jason O'Neil]

Hey @MarkStosberg, thanks for investigating. I already had a linked file at /etc/rc5.d/S02apache2. So it does attempt to start, but it is failing. I'll update my question. Thanks though

[David Foerster]

Welcome to Ask Ubuntu! I recommend to edit this answer to expand it with specific details about what this command does and what it is supposed to achieve. (See also How do I write a good answer? for general advice about what sorts of answers are considered most valuable on AskUbuntu.)