Having trouble running supervisord using systemctl (systemd)

Question

I am unable to get supervisor to successfully run via systemctl.

I am running Centos 7.5

I install supervisor with sudo pip install supervisor

$ supervisor -v: 3.3.4

$ which supervisord: /usr/bin/supervisord

I put my config file in /etc/supervisor/supervisord.conf (0644 and root:root for the file and the supervisor folder) The config file contains the following (I removed commented out lines for brevity):

[supervisord]
logfile=/home/myuser/logs/supervisord.log ;
logfile_maxbytes=50MB       ; (max main logfile bytes b4 rotation;default 50MB)
logfile_backups=10          ; (num of main logfile rotation backups;default 10)
loglevel=info               ; (logging level;default info; others: debug,warn)
pidfile=/var/run/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
nodaemon=false              ; (start in foreground if true;default false)
minfds=1024                 ; (min. avail startup file descriptors;default 1024)
minprocs=200                ; (min. avail process descriptors;default 200)
environment=WEBAPP_CONFIG="testing"

[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

[supervisorctl]
serverurl=unix:///var/tmp/supervisor.sock ; use a unix:// URL  for a unix socket

[inet_http_server] ; inet (TCP) server disabled by default
port=127.0.0.1:9001 ; (ip_address:port specifier, *:port for all iface)

[unix_http_server]
file = /var/tmp/supervisor.sock
chmod = 0777

[program:redisgo]
command=/usr/bin/redis-server /etc/redis.conf
autostart=true
autorestart=unexpected
stdout_logfile=/home/myuser/logs/redis.log
stderr_logfile=/home/myuser/logs/rediserr.log

[program:celery]
command = /home/myuser/mydevelopment/git/myproj/env/bin/celery worker -A webapp.celery --loglevel=info
directory = /home/myuser/mydevelopment/git/my_proj
numprocs=1
stdout_logfile=/home/myuser/logs/celeryworker.log
stderr_logfile=/home/myuser/logs/celeryworkererr.log
autostart=true
autorestart=unexpected
startsecs=10
user=myuser
stopwaitsecs = 600

; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true

; if rabbitmq is supervised, set its priority higher
; so it starts first
priority=998

[program:datacollect]
command = /home/myuser/mydevelopment/git/myproj/env/bin/python /home/myuser/mydevelopment/git/myproj/data_monitoring/collection_manager.py
stdout_logfile=/home/myuser/logs/datacollect.log
stderr_logfile=/home/myuser/logs/datacollecterr.log
autostart=true
autorestart=unexpected

# BEGIN ANSIBLE MANAGED BLOCK

[program:cloud_sql_proxy]
environment=GOOGLE_APPLICATION_CREDENTIALS="/home/myuser/credentials/gcp_service_account.json"
command=/usr/local/bin/cloud_sql_proxy -instances=my-proj:us-central1:mysql-proj=tcp:3306
stdout_logfile=/home/myuser/logs/cloud_proxy.log
stderr_logfile=/home/myuser/logs/cloud_proxyerr.log
autostart=true
autorestart=true
# END ANSIBLE MANAGED BLOCK

I put a systemd file called supervisord.service in /usr/lib/systemd/system/supervisord.service (again, 0644 and root:root) that contains the following:

[Unit]
Description=Supervisor process control system for UNIX
Documentation=http://supervisord.org
After=network.target

[Service]
ExecStart=/usr/bin/supervisord -c /etc/supervisor/supervisord.conf
ExecStop=/usr/bin/supervisorctl $OPTIONS shutdown
ExecReload=/usr/bin/supervisorctl -c /etc/supervisor/supervisord.conf $OPTIONS reload
KillMode=process
Restart=on-failure
RestartSec=50s

[Install]
WantedBy=multi-user.target

When I start supervisord via $ sudo systemctl start supervisord I see the following issues in the supervisord.log:

2018-07-19 19:26:07,655 CRIT Supervisor is running as root.  Privileges were not dropped because no user is specified in the config file.  If you intend to run as root, you can set user=root in the config file to avoid this message.
2018-07-19 19:26:07,655 WARN For [program:cloud_sql_proxy], redirect_stderr=true but stderr_logfile has also been set to a filename, the filename has been ignored
2018-07-19 19:26:07,663 INFO RPC interface 'supervisor' initialized
2018-07-19 19:26:07,663 CRIT Server 'inet_http_server' running without any HTTP authentication checking
2018-07-19 19:26:07,664 INFO RPC interface 'supervisor' initialized
2018-07-19 19:26:07,664 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2018-07-19 19:26:07,666 INFO daemonizing the supervisord process
2018-07-19 19:26:07,667 INFO supervisord started with pid 22196
2018-07-19 19:26:07,752 INFO spawned: 'celery' with pid 22200
2018-07-19 19:26:07,753 INFO spawned: 'cloud_sql_proxy' with pid 22201
2018-07-19 19:26:07,754 INFO spawned: 'redisgo' with pid 22202
2018-07-19 19:26:07,756 INFO spawned: 'datacollect' with pid 22203
2018-07-19 19:26:07,758 INFO waiting for celery, cloud_sql_proxy, redisgo, datacollect to die
2018-07-19 19:26:08,831 INFO success: cloud_sql_proxy entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-07-19 19:26:08,831 INFO success: redisgo entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-07-19 19:26:11,092 INFO waiting for celery, cloud_sql_proxy, redisgo, datacollect to die
2018-07-19 19:26:14,294 INFO waiting for celery, cloud_sql_proxy, redisgo, datacollect to die
2018-07-19 19:26:18,118 INFO success: celery entered RUNNING state, process has stayed up for > than 10 seconds (startsecs)
2018-07-19 19:26:18,118 WARN killing 'datacollect' (22203) with SIGKILL
2018-07-19 19:26:18,118 INFO waiting for celery, cloud_sql_proxy, redisgo, datacollect to die
2018-07-19 19:26:18,124 INFO stopped: datacollect (terminated by SIGKILL)
2018-07-19 19:26:18,198 INFO stopped: redisgo (exit status 0)
2018-07-19 19:26:18,199 INFO stopped: cloud_sql_proxy (terminated by SIGTERM)
2018-07-19 19:26:21,149 INFO waiting for celery to die
2018-07-19 19:26:24,157 INFO waiting for celery to die
...

That last line prints several times and then celery finally dies. It's strange because I can't see what is causing supervisor to kill everything and wait for all the process to die

If I start superviosrd with $ sudo supervisord, then supervisor will successfully start. However, it is not registering with systemctl because $ systemctl status supervisord gives:

● supervisord.service - Supervisor process control system for UNIX
   Loaded: loaded (/usr/lib/systemd/system/supervisord.service; enabled; vendor preset: disabled)
   Active: inactive (dead) since Thu 2018-07-19 19:26:07 UTC; 4min 8s ago
     Docs: http://supervisord.org
 Main PID: 22193 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/supervisord.service

Even though it is running because supervisorctl works and $ ps -aux | grep super shows:

myuser   22185  0.0  0.0 107988   612 pts/0    S+   19:25   0:00 tail -f supervisord.log
root     22230  0.0  1.1 220796 11256 ?        Ss   19:28   0:00 /usr/bin/python2 /bin/supervisord
myuser   22282  0.0  0.0 112704   972 pts/3    R+   19:31   0:00 grep --color=auto super

This doesn't work though because I need systemctl to manage it so that it starts on boot. Can you see any issues?

Solution

You should change supervisord.service.
Add the following on the Service section:

[Service]
Type=forking

Have a look at systemd documentation:
https://www.freedesktop.org/software/systemd/man/systemd.service.html
And the clearest explanation you'll find over internet on the arch-wiki:
https://wiki.archlinux.org/index.php/Systemd#Service_types
There you will see the differences and know why it stops just after start. The default mode is simple and it won't work with this kind of service.
Cheers!

Comments

[Michael Hampton]

This seems utterly pointless. There's no need for supervisor on a systemd system; you can just use a systemd unit to start your service directly. (Celery example) Not to mention that EPEL already includes supervisor for people who need it to run old stuff.

[Brian Leach]

Not going to argue that it isn't convoluted. But, there is a lot of infrastructure around it that doesn't need to be reinvented. Any input on the issue?