Apache will not start, no httpd error messages

Question

I am trying to start the httpd service on RHEL 7.

When I run systemctl start httpd, it fails.

Here is the output of journalctl -xe. There are no apparent error message for httpd, Would the errors for polkitd be affecting apache? How would I resolve these?

Jun 10 10:59:50 localhost.localdomain polkitd[850]: Registered Authentication Agent for unix-process:4462:116143 (system bus name :1.57 [/usr/bin/pkttyagent --notify-fd 5 --fallback], object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8)
Jun 10 10:59:50 localhost.localdomain systemd[1]: Starting The Apache HTTP Server...
-- Subject: Unit httpd.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit httpd.service has begun starting up. 
Jun 10 10:59:50 localhost.localdomain systemd[1]: httpd.service: main process exited, code=exited, status=1/FAILURE
Jun 10 10:59:50 localhost.localdomain kill[4468]: kill: cannot find process ""
Jun 10 10:59:50 localhost.localdomain systemd[1]: httpd.service: control process exited, code=exited status=1
Jun 10 10:59:50 localhost.localdomain systemd[1]: Failed to start The Apache HTTP Server.
-- Subject: Unit httpd.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit httpd.service has failed.
--
-- The result is failed.
Jun 10 10:59:50 localhost.localdomain systemd[1]: Unit httpd.service entered failed state.
Jun 10 10:59:50 localhost.localdomain systemd[1]: httpd.service failed.
Jun 10 10:59:50 localhost.localdomain polkitd[850]: Unregistered Authentication Agent for unix-process:4462:116143 (system bus name :1.57, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (disconnected from bus)

EDIT: log output and additional info

Here is the output of tail -f 10 /var/log/httpd/error_log. The log files are stored on an NFS mounted directory (hosted remotely). I can edit the files from the machine running apache, but apache itself doesn't seem to be able to write to them.

==> /var/log/httpd/error_log <==
(13)Permission denied: AH00091: httpd: could not open error log file /data/web/logs/new-error.log.
AH00015: Unable to open logs
(13)Permission denied: AH00091: httpd: could not open error log file /data/web/logs/new-error.log.
AH00015: Unable to open logs
(13)Permission denied: AH00091: httpd: could not open error log file /data/web/logs/new-error.log.
AH00015: Unable to open logs
(13)Permission denied: AH00091: httpd: could not open error log file /data/web/logs/new-error.log.
AH00015: Unable to open logs
(13)Permission denied: AH00091: httpd: could not open error log file /data/web/logs/io-new-error.log.
AH00015: Unable to open logs

EDIT 2: Additional logs

[root@localhost web]# sealert -a /var/log/audit/audit.log
 78% donetype=AVC msg=audit(1465576122.933:606): avc:  denied  { write } for  pid=4384 comm="httpd" name="logs" dev="0:38" ino=1060076 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:nfs_t:s0 tclass=dir

**** Invalid AVC allowed in current policy ***

type=AVC msg=audit(1465576695.541:615): avc:  denied  { read } for  pid=4489 comm="httpd" name="sims" dev="0:38" ino=1060159 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:nfs_t:s0 tclass=lnk_file

**** Invalid AVC allowed in current policy ***

 78% donetype=AVC msg=audit(1465576695.580:616): avc:  denied  { read } for  pid=4489 comm="httpd" name="sims" dev="0:38" ino=1060159 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:nfs_t:s0 tclass=lnk_file

**** Invalid AVC allowed in current policy ***

type=AVC msg=audit(1465576695.581:617): avc:  denied  { write } for  pid=4489 comm="httpd" name="logs" dev="0:38" ino=1060076 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:nfs_t:s0 tclass=dir

**** Invalid AVC allowed in current policy ***

100% done
found 0 alerts in /var/log/audit/audit.log

Solution

If your RHEL 7 system has SELinux enforcing, you need to enable the SELinux boolean values to get NFS working. To check if your system is utilizing SELinux:

# getenforce

If the above returns enforcing, please continue:

# getsebool -a | grep httpd_use_nfs

If the above returns httpd_use_nfs --> off, run the following command to permanently allow Apache to use NFS.

# setsebool -P httpd_use_nfs on

Apache should be allowed to write to those error log files now.

Comments

[clhy]

What changed did you made in httpd.conf ? Show me the output of tail -f 10 /var/log/httpd/error_log

[Matt Pennington]

I added the log output above. It looks like there is a problem writing to the log files. I haven't made any changes directly to httpd.conf

[Matt Pennington]

Thanks Jeffrey. I've set httpd_use_nfs to on, but I'm still getting the error Permission denied: AH00091: httpd: could not open error log file. Do the files need to have a particular permissions level or owner?

[Jeffrey Wen]

Let's confirm what user & group Apache is running as: httpd -S | grep 'User\|Group'. For example, let's say you get User: name="apache" and Group: name="apache", then your NFS directory /data/web/logs should be mounted with user & group as apache:apache

[Matt Pennington]

The user:group is indeed apache:apache. I chown'd the /data/web/logs directory to apache:apache, but I'm still getting the same errors. do the actual log files needto be apache:apache too?

[Jeffrey Wen]

Yes, the logs should be owned by apache:apache too. You have two options- Option 1: Delete the logs (if they are useless) and restart apache; apache will re-create the logs automatically. Option 2: chown all logs to apache:apache

[Matt Pennington]

I tried Option 1 and Option 2 with no luck. With Option 1 I got the same errors and it did not recreate the log files

[Jeffrey Wen]

Are the apache logs still the same? If so, could you see if SELinux is still giving you errors? sealert -a /var/log/audit/audit.log

[Matt Pennington]

I posted the output of the audit.log in the OP. I tried to apply the fix from this issue: serverfault.com/a/677805, but it didn't have any effect.

[Jeffrey Wen]

Let's try to separate the issue first. Can you try changing the default Apache logs back to /var/log/httpd/{access_log/error_log} and then restart Apache to see if {access_log/error_log} gets re-generated? If your logs get re-generated and Apache works, at least we can focus more on the NFS part.

[Matt Pennington]

I commented out the log and httpd starts fine