CentOS 7 boots too fast and network is not ready when executing cron scripts

Question

I have just upgraded from CentOS 6.5 to 7.0 and I am not too happy as the new systemd is probably giving me problems. It seems it is just simply booting too fast, starting up processes asynchronously and screwing up service dependencies.

For example I have a few scripts setup in crond which are triggered after a reboot:

@reboot    /root/scripts/check_gmail.sh
@reboot    /root/scripts/start_gps_listener.sh

This results in all kind of weird errors (only showing one of them):

Warning: stream_socket_client(): unable to connect to tcp://192.168.20.4:4001 
  (Network is unreachable) in /root/scripts/check_gmail.php on line 137
  ERROR: Network is unreachable (101)

In the above I am writing to a TCP socket. It is pretty clear to me that crond is started before the network is properly initialized as the network is unreachable.

The same thing goes with Apache and MySQL (MariaDB). MySQL is quite slow to startup (much data) meaning that both Apache and a lot of my crond startup scripts are failing as the MySQL database is not running when the scripts are being called.

I have tried to setup dependencies but without any luck; I have appended network and mysql services to [Unit] (as seen with systemctl list-dependencies). Ideally all services wait until MySQL is up and running:

vi /lib/systemd/system/httpd.service
  [Unit]
  Description=The Apache HTTP Server
  After=network.target remote-fs.target nss-lookup.target network.service mysql.service

vi /lib/systemd/system/crond.service
  [Unit]
  Description=Command Scheduler
  After=syslog.target auditd.service systemd-user-sessions.service time-sync.target network.service mysql.service

When booting with the above I get the same errors. I also get the emails in mailq as the network/DNS is not ready when processing cron-scripts. A few minutes after startup they are sent correctly.

Can anyone help getting this right by making sure the services are fired in the correct order? It seems very wrong that it is so fast booting and ideally it did it the good old way, "launching one serice... wait... launching a new service... wait... so on).

Do note that I am not sure it is systemd that is my problem - it is just my theory of what I can read from the net.

Solution

After a lot more reading I found the solution that works for me.

I read this guide, Running Services After the Network is up. A little quote from the guide:

This will ensure that all configured network devices are up and have an IP address assigned before boot continues.

This is exactly what I wanted so I enabled this service and set the a dependency rule in the service file for crond:

[root@srv]# systemctl enable NetworkManager-wait-online

[root@srv]# vi /lib/systemd/system/crond.service
  Requires=network.target
  After=syslog.target auditd.service systemd-user-sessions.service time-sync.target network.target mysqld.service

As mysqld is still based on the old init.d I needed to create a systemd service as suggested here, systemctl enable differs from systemctl start:

[root@srv]# vi /lib/systemd/system/mysqld.service
  [Unit]
  Description=MySQL Server
  After=network.target
  [Service]
  Type=forking
  ExecStart=/etc/rc.d/init.d/mysql start
  ExecStop=/etc/rc.d/init.d/mysql stop
  [Install]
  WantedBy=multi-user.target

[root@srv]# systemctl daemon-reload
[root@srv]# chkconfig mysql off
[root@srv]# systemctl enable mysqld

And finally setup the Apache service to startup after MySQL:

[root@srv]# vi /lib/systemd/system/httpd.service
  Requires=mysqld.service
  After=network.target remote-fs.target nss-lookup.target mysqld.service

This works for me at least.

I have used these commands to check it afterwards where I clearly can see that network is started up prior to at least MySQL and Apache. I though cannot see the crond anywhere but I can see it is working in my scripts:

[root@srv]# systemd-analyze critical-chain
  multi-user.target @10.510s
    + httpd.service @10.344s +165ms
      + mysqld.service @9.277s +1.065s
        + network.target @9.273s
          + network.service @8.917s +355ms
            + iptables.service @444ms +157ms
              + basic.target @443ms
                [CUT]

A couple of other useful commands I used are:

# See exactly what takes how long (who to blame for the delay)
[root@srv]# systemd-analyze blame

# Check available names that can be used in the service files
[root@srv]# systemctl list-unit-files

If any one can see any better way to do this then please share.

Comments

[terdon]

Could you post the output of grep -i concurrency /etc/default/rcS? I might be mixing up my init systems but I seem to recall that controls whether processes wait for one another to finish.

[terdon]

Sorry, I don't know where the CentOS equivalent would be. I was thinking of what is described here for Debian, which makes services start in parallel. There might be something similar set in your case.

[casey]

Try adding Requires=network.target to the units above.

[Brian Topping]

+1 for posting the commands you used to debug this. I was able to resolve a really nasty knot of problems with systemd-analyze critical-chain. Not only am I going to use that often, but I am suddenly sold on systemd. Thanks!

[Ludovic Ronsin]

You should not modify services files managed by your distribution package manager. Instead you'd better use drop-in configuration files. See the answer to How do I override or configure systemd services ?