Redis Sentinel Systemd services failing to start on Ubuntu 16.04 and CentOS 7

Question

Having trouble getting my Redis sentinel systemd services to start.

I can run sentinels manually with success -- e.g.:

root@poolwc:/etc/redis# redis-sentinel ./sentinel.conf --sentinel

Three redis servers at 10.9.8.1, .2, .3

.2 and .3 are slaves to .1 and replication is working

redis service user has 0600 r/w access to both redis.conf and sentinel.conf for all three hosts

For Slave1 and Slave2 (10.9.8.2 and 10.9.8.3) I have the following added to redis.conf:

+ slaveof 127.0.0.1 6380

For Master, Slave1, and Slave2:

sentinel.conf:

sentinel monitor mymaster 10.9.8.1 6379 2
sentinel down-after-milliseconds mymaster 60000
sentinel failover-timeout mymaster 180000
sentinel parallel-syncs mymaster 1

Output from master on service start:

root@poolwc:/etc/redis# systemctl start redis-sentinel
Job for redis-sentinel.service failed because the control process exited with error code. See "systemctl status redis-sentinel.service" and "journalctl -xe" for details.

    root@poolwc:/etc/redis# systemctl status redis-sentinel.service
    ● redis-sentinel.service - Advanced key-value store
       Loaded: loaded (/lib/systemd/system/redis-sentinel.service; enabled; vendor preset: enabled)
       Active: inactive (dead) (Result: exit-code) since Mon 2018-02-26 21:54:12 

PST; 6s ago Docs: http://redis.io/documentation, man:redis-sentinel(1) Process: 5376 ExecStopPost=/bin/run-parts --verbose /etc/redis/redis-sentinel.post-down.d (code=exited, status=0/SUCCESS) Process: 7893 ExecStart=/usr/bin/redis-sentinel /etc/redis/sentinel.conf (code=exited, status=1/FAILURE) Process: 7890 ExecStartPre=/bin/run-parts --verbose /etc/redis/redis-sentinel.pre-up.d (code=exited, status=0/SUCCESS)

Feb 26 21:54:12 poolwc systemd[1]: redis-sentinel.service: Control process exited, code=exited status=1
Feb 26 21:54:12 poolwc systemd[1]: Failed to start Advanced key-value store.
Feb 26 21:54:12 poolwc systemd[1]: redis-sentinel.service: Unit entered failed state.
Feb 26 21:54:12 poolwc systemd[1]: redis-sentinel.service: Failed with result 'exit-code'.
Feb 26 21:54:12 poolwc systemd[1]: redis-sentinel.service: Service hold-off time over, scheduling restart.
Feb 26 21:54:12 poolwc systemd[1]: Stopped Advanced key-value store.
Feb 26 21:54:12 poolwc systemd[1]: redis-sentinel.service: Start request repeated too quickly.
Feb 26 21:54:12 poolwc systemd[1]: Failed to start Advanced key-value store.

Output from Slave2 on service start -- running, but why the exit on failure?:

[root@ns1 etc]# systemctl start redis-sentinel
[root@ns1 etc]# systemctl status redis-sentinel
● redis-sentinel.service - Redis Sentinel
   Loaded: loaded (/usr/lib/systemd/system/redis-sentinel.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/redis-sentinel.service.d
           └─limit.conf
   Active: active (running) since Mon 2018-02-26 21:55:58 PST; 5s ago
  Process: 12876 ExecStop=/usr/libexec/redis-shutdown redis-sentinel (code=exited, status=1/FAILURE)
 Main PID: 12944 (redis-sentinel)
   CGroup: /system.slice/redis-sentinel.service
           └─12944 /usr/bin/redis-sentinel *:26379 [sentinel]

Feb 26 21:55:58 ns1 redis-sentinel[12944]: |    `-._`-._        _.-'_.-'    |           http://redis.io
Feb 26 21:55:58 ns1 redis-sentinel[12944]: `-._    `-._`-.__.-'_.-'    _.-'
Feb 26 21:55:58 ns1 redis-sentinel[12944]: |`-._`-._    `-.__.-'    _.-'_.-'|
Feb 26 21:55:58 ns1 redis-sentinel[12944]: |    `-._`-._        _.-'_.-'    |
Feb 26 21:55:58 ns1 redis-sentinel[12944]: `-._    `-._`-.__.-'_.-'    _.-'
Feb 26 21:55:58 ns1 redis-sentinel[12944]: `-._    `-.__.-'    _.-'
Feb 26 21:55:58 ns1 redis-sentinel[12944]: `-._        _.-'
Feb 26 21:55:58 ns1 redis-sentinel[12944]: `-.__.-'
Feb 26 21:55:58 ns1 redis-sentinel[12944]: 12944:X 26 Feb 21:55:58.490 # Sentinel ID is 659761a6f8328e59d43086ef3265450fcec5d2f5
Feb 26 21:55:58 ns1 redis-sentinel[12944]: 12944:X 26 Feb 21:55:58.490 # +monitor master mymaster 10.9.8.1 6379 quorum 2

Output from Slave2 on service start:

root@poolec:/etc/redis# systemctl start redis-sentinel
Job for redis-sentinel.service failed because the control process exited with error code. See "systemctl status redis-sentinel.service" and "journalctl -xe" for details.
root@poolec:/etc/redis# systemctl status redis-sentinel

    ● redis-sentinel.service - Advanced key-value store
   Loaded: loaded (/lib/systemd/system/redis-sentinel.service; enabled; vendor preset: enabled)
   Active: inactive (dead) (Result: exit-code) since Tue 2018-02-27 05:57:09 UTC; 5s ago
     Docs: http://redis.io/documentation,
           man:redis-sentinel(1)
  Process: 4130 ExecStart=/usr/bin/redis-sentinel /etc/redis/sentinel.conf (code=exited, status=1/FAILURE)
  Process: 4127 ExecStartPre=/bin/run-parts --verbose /etc/redis/redis-sentinel.pre-up.d (code=exited, status=0/SUCCESS)

Feb 27 05:57:09 poolec systemd[1]: redis-sentinel.service: Unit entered failed state.
Feb 27 05:57:09 poolec systemd[1]: redis-sentinel.service: Failed with result 'exit-code'.
Feb 27 05:57:09 poolec systemd[1]: redis-sentinel.service: Service hold-off time over, scheduling restart.
Feb 27 05:57:09 poolec systemd[1]: Stopped Advanced key-value store.
Feb 27 05:57:09 poolec systemd[1]: redis-sentinel.service: Start request repeated too quickly.
Feb 27 05:57:09 poolec systemd[1]: Failed to start Advanced key-value store.

None of them leave any output in /var/log/redis/

Any ideas?

Thanks very much!

Solution

Does appear to be solved by the following, Found here: Redis not starting with systemctl

See the configuration file:

If you run Redis from upstart or systemd, Redis can interact with your supervision tree. Options: supervised no - no supervision interaction supervised upstart - signal upstart by putting Redis into SIGSTOP mode supervised systemd - signal systemd by writing READY=1 to $NOTIFY_SOCKET supervised auto - detect upstart or systemd method based on UPSTART_JOB or NOTIFY_SOCKET environment variables Note: these supervision methods only signal "process is ready." They do not enable continuous liveness pings back to your supervisor.

supervised no

Needs to be changed to:

supervised systemd

You can also pass this on the command line, which overrides the setting in redis.conf. Red Hat based systems do this. This also allows for running the same redis instance manually or from systemd without changing the config file.

ExecStart=/usr/bin/redis-server /etc/redis.conf --supervised systemd In addition, you also need to tell systemd that redis will be operating in this mode by setting Type=notify in the [Service] section.

Comments

[Admin]

We are also looking into this problem, and have the same issue. If we get an answer ill update, anyone else with an insight, please feel free to inject suggestions. We are looking into whether it is the user permission, since the systemd starts the service as the specified user.