httpd not starting because /run/httpd is missing

Question

httpd fails to start on one of my servers running centos 7 because /run/httpd is missing, which means httpd can't create /run/httpd/httpd.pid and thus fails to start.

My question is, why is /run/httpd being deleted. I am not deleting the dir.

Running httpd-2.4.6-18.el7.centos.x86_64 from base CentOS repo

Solution

I had a similar problem, and it turned out to be a systemd issue... not as in a problem with systemd, but as in my not understanding how to work with it.

So, /run used to be in /var/run, and it used to be more or less persistent (if I understand correctly). Now, it is often in a fstmp file system, so a reboot removes everything in /run. This can be corrected by telling systemd that the service needs a /run dir thus:

My tomcat.service files failed when like this:

[Unit]
Description=Apache Tomcat Web Application Container
After=syslog.target network.target

[Service]
Type=forking

Environment=CATALINA_PID=/run/tomcat/tomcat.pid
Environment=CATALINA_HOME=/usr/share/tomcat8/
Environment=CATALINA_BASE=/usr/share/tomcat8/
Environment='CATALINA_OPTS=-server -XX:+UseParallelGC -Dsun.net.inetaddr.ttl=30'
Environment='JAVA_OPTS=-Djava.awt.headless=true -Djava.security.egd=file:/dev/./urandom'

ExecStart=/usr/share/tomcat8/bin/startup.sh
ExecStop=/bin/kill -15 $MAINPID
PIDFile=/run/tomcat/tomcat.pid
SuccessExitStatus=143

User=tomcat
Group=tomcat

[Install]
WantedBy=multi-user.target

It worked when like this:

[Unit]
Description=Apache Tomcat Web Application Container
After=syslog.target network.target

[Service]
Type=forking

Environment=CATALINA_PID=/run/tomcat/tomcat.pid
Environment=CATALINA_HOME=/usr/share/tomcat8/
Environment=CATALINA_BASE=/usr/share/tomcat8/
Environment='CATALINA_OPTS=-server -XX:+UseParallelGC -Dsun.net.inetaddr.ttl=30'
Environment='JAVA_OPTS=-Djava.awt.headless=true -Djava.security.egd=file:/dev/./urandom'

RuntimeDirectory=tomcat
RuntimeDirectoryMode=775
ExecStart=/usr/share/tomcat8/bin/startup.sh
ExecStop=/bin/kill -15 $MAINPID
PIDFile=/run/tomcat/tomcat.pid
SuccessExitStatus=143

User=tomcat
Group=tomcat

[Install]
WantedBy=multi-user.target

I only needed to do the above becuase we "rolled our own" tomcat in order to get a newer version of it.

However, we sometimes had the same error with apache, but with a different cause: Apparently, the trouble came down to the fact that we bound to an ldap server prior to installing httpd, and the ldap server has an apache user and group. When the apache install scripts saw the user and group, it did not create the local user and group. However, the ldap user and group would sometimes not be available (slow network?) when a systemd-tmpfiles-setup.service service runs at boot time. That service is a different way to create the runtime dir... So we forced a local apache user and group (with the same uid/gid as listed in ldap), and everything is golden.

I am in no way an expert in any of this, I am mostly documenting what our senior guy found while trying to orchestrate apache/tomcat apps with Puppet. I hope this helps someone.

There are other post related to this one: Apache's PidFile directory is removed every boot

Comments

[quadruplebucky]

stackoverflow.com/questions/16156761/…