What would likely cause directories in /var/run to disappear / not be created on startup? (CentOS 7.3)

Question

Ran into an issue today where Apache would not load because /var/run/httpd was missing. After much hunting to find that cause of the error I was getting ([Tue Jul 25 00:06:31.574386 2017] [auth_digest:error] [pid 6179] (2)No such file or directory: AH01760: failed to initialize shm - all nonce-count checking, one-time nonces, and MD5-sess algorithm disabled).

Then I saw that MariaDB was missing its /var/run/mysql directory, and PostgreSQL was missing its directory.

What would cause those to not be made on system start-up / saved between reboots?

I would not expect to need to modify init scripts as suggested in this question.

Edit: outputs of ls -dl /var/run & ls -dl /run

lrwxrwxrwx. 1 root root 6 Dec 15  2015 /var/run -> ../run
drwxr-xr-x 25 root root 800 Jul 26 03:40 /run

Solution 1

Not a fix - but a fix.

After trawling through other SE network sites, forums, newsgroups, and the like, I wondered if it might be related to mount order for local and remote file systems (there are a pair of CIFS shares mounted on this server, which each have subdirectories bind mounted elsewhere).

I pushed the CIFS shares higher in /etc/fstab, and moved all bind mounts to the bottom.

Then I rebooted.

Now /run aka /var/run is working as expected.

I don't know that editing /etc/fstab mattered - but it's the only substantive change made that I know of.

Solution 2

{/var,}/run is vaporized by systemd on boot. It's a wonderful and entirely unexpected surprise with such low value.

If you want to ensure a directory exists after boot, you'll have to hook something in with the tmpfiles section of systemd.

/usr/lib/tmpfiles.d/httpd.conf
::::::::::::::
d /run/httpd   710 root apache
d /run/httpd/htcacheclean   700 apache apache

/usr/lib/tmpfiles.d/mariadb.conf
::::::::::::::
d /var/run/mariadb 0755 mysql mysql -

And, if your tmpfiles entries are missing, I'm betting you hand-roll your own packages (and tune, and test, and promote, for every.single.release). As long as we agree it's usually a bad idea for anything with scale or an SLA, that's cool though.

Solution 3

These config files reside /usr/lib/tmpfiles.d and /etc/tmpfiles.d/.

I needed to unmask and enable the systemd-tmpfiles-setup.service in order to have /run automatically setup at boot: systemctl unmask systemd-tmpfiles-setup.service systemctl enable systemd-tmpfiles-setup.service

Comments

[Rui F Ribeiro]

something does not add up. was your var filled up and you took care of it? are you low in memory? is run or /var/run in disk or ram?

[Rui F Ribeiro]

add also the output of free -m please

[warren]

@bbb31 - tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)