Apache2 Fails to start after recent update to 2.4.34, no clue why

Question

At 7:57:18PM last night, July 17th Apache2 shut down and quit working. I didn't notice until earlier today (the 18th.)

Updates Occurred: /var/log/apt/history.log shows that there were 3 updates performed:

Start-Date: 2018-07-17  19:57:18
Commandline: apt-get -y install apache2 Install: libjansson4:amd64 
(2.7-3, automatic) Upgrade: apache2-data:amd64 (2.4.33- 
3.0+ubuntu16.04.1+deb.sury.org+1, 2.4.34- 
1+ubuntu16.04.1+deb.sury.org+5), apache2-bin:amd64 (2.4.33- 
3.0+ubuntu16.04.1+deb.sury.org+1, 2.4.34 
1+ubuntu16.04.1+deb.sury.org+5), apache2:amd64 (2.4.33 
3.0+ubuntu16.04.1+deb.sury.org$ End-Date: 2018-07-17  19:57:23

Start-Date: 2018-07-17  19:57:37
Commandline: apt-get -y install apache2-doc
Upgrade: apache2-doc:amd64 (2.4.33-3.0+ubuntu16.04.1+deb.sury.org+1, 
2.4.34-1+ubuntu16.04.1+deb.sury.org+5)
End-Date: 2018-07-17  19:57:39

Start-Date: 2018-07-17  19:57:43
Commandline: apt-get -y install apache2-suexec-custom
Upgrade: apache2-suexec-custom:amd64 (2.4.33- 
3.0+ubuntu16.04.1+deb.sury.org+1, 2.4.34- 
1+ubuntu16.04.1+deb.sury.org+5)
End-Date: 2018-07-17  19:57:44

Upon Entering: /etc/init.d/apache2 start

I get: Job for apache2.service failed because the control process exited with error code. See "systemctl status apache2.service" and "journalctl -xe" for details. failed!

systemctl status apache2.service yields:

apache2.service - The Apache HTTP Server
   Loaded: loaded (/lib/systemd/system/apache2.service; enabled; 
vendor preset: enabled)
   Active: failed (Result: exit-code) since Wed 2018-07-18 15:01:46 
MST; 3min 50s ago
  Process: 8835 ExecStop=/usr/sbin/apachectl stop (code=exited, 
status=0/SUCCESS)
  Process: 21710 ExecStart=/usr/sbin/apachectl start (code=exited, 
status=1/FAILURE)
 Main PID: 8832 (code=exited, status=1/FAILURE)

journalctl -xe gives me this:

Jul 18 15:20:04 ns1.[masked].com apachectl[23823]: apache2: Syntax 
error on line 146 of /etc/apache2/apache2.conf: Syntax error on line 
2 of /etc/apache2/mods-enabled/lbmethod_byrequests.load:

Line 146 of /etc/apache2/apache2.conf reads:

DefaultRuntimeDir ${APACHE_RUN_DIR}

and line 2 of /etc/apache2/mods-enabled/lbmethod_byrequests.load:

LoadModule lbmethod_byrequests_module /usr/lib/apache2/modules/mod_lbmethod_byrequests.so

At this point, I'm completely lost. Line 146 just references the .load file which points to a path to a file that exists...so I don't know what the syntax error is on line 2 of lbmethod_byrequests.load.

Anyone else having trouble with Apache after the update to 2.4.34?

Solution 1

You're missing some of the error. The full error is:

apache2: Syntax error on line 146 of /etc/apache2/apache2.conf: Syntax error 
on line 2 of /etc/apache2/mods-enabled/lbmethod_byrequests.load: Cannot load 
/usr/lib/apache2/modules/mod_lbmethod_byrequests.so into server: 
/usr/lib/apache2/modules/mod_lbmethod_byrequests.so: undefined symbol: 
ap_proxy_balancer_get_best_worker

It appears to be caused by module load order. lbmethod_byrequests depends on proxy but lbmethod_byrequests is being loaded first, leading to an undefined symbol error that proxy would provide if it were loaded.

If you mv /etc/apache2/mods-enabled/lbmethod_byrequests.load /etc/apache2/mods-enabled/zzlbmethod_byrequests.load, Apache starts normally. Unfortunately I can't say why this happens now and didn't before.

Solution 2

I had to change the module load order in the proxy.conf in a similar scenario:

I'm running the alpine apache2-proxy package. It fails with:

cannot load modules/mod_lbmethod_bybusyness.so into server: 
Error relocating /var/www/modules/mod_lbmethod_bybusyness.so: 
ap_proxy_balancer_get_best_worker: symbol not found 

The guys here were talking about a bug being fixed but the load order still being an issue. I'm not 100% clear on the details but in my instance the proxy config was in a different place:

etc/apache2/conf.d/proxy.conf

Here I made sure mod_proxy.so was listed before mod_lbmethod_byrequests Problem solved. Only took me half a day.

Comments

[Jon Griffith]

OH MY GOD! I didn't even see your answer until just now...GARGHLHGLDH! So much for notifications ;). So the ZZ just alphabetizes the file so it fires off last...whatever it's doing. IT WORKED How did you figure that out? I did see the entire error, eventually.

[Jon Griffith]

All of my certificates for all of my sites seem to be trashed, however.

[Jon Griffith]

Can I buy you lunch or something? You just saved my butt.