/etc/inittab respawn script migrating from RHEL/CentOS 5.x to 6.x

Question

I have a non-forking perl script running as a TCP-sockets daemon (this script is a backend for a multiplayer game) in CentOS 5.7. It is being started and respawned by /etc/inittab:

pref:3:respawn:/bin/su -c '/usr/local/pref/pref.pl >/tmp/pref-`date +%a`.txt 2>&1' afarber

It is being restarted every night by the cronjob:

33    1     *     *     *     kill `cat /tmp/pref.pid`

(where the /tmp/pref.pid file is created by the script itself).

This setup has worked well for me since many moons. Now I'm trying to upgrade to CentOS 6.x and have created a new /etc/init/pref.conf file after reading "man 5 init":

start on stopped rc RUNLEVEL=3
stop on starting rc RUNLEVEL=[!3]
console output
respawn
chdir /tmp
exec /bin/su -c '/usr/local/pref/pref.pl >/tmp/pref-`date +%a`.txt 2>&1' afarber

And can start it with

# sudo initctl start pref
pref start/running, process 2590

and also see the script running under user afarber with "ps uawx" and listening at port 8080 (as it should) with "netstat -an".

But my problem is that I can't stop or restart the script (and I need that for the nightly cronjob):

# sudo initctl restart pref
initctl: Unknown instance:

# sudo initctl stop pref
initctl: Unknown instance:

Any ideas please?

(And I don't want to install any 3rd party software, like daemontools/Tivoli/etc. - because I want to be my web server to be easily reinstallable and movable to other hosters).

UPDATE: here is what I see -

# initctl reload-configuration

# initctl list
rc stop/waiting
tty (/dev/tty3) start/running, process 1515
tty (/dev/tty2) start/running, process 1513
tty (/dev/tty1) start/running, process 1511
tty (/dev/tty6) start/running, process 1521
tty (/dev/tty5) start/running, process 1519
tty (/dev/tty4) start/running, process 1517
plymouth-shutdown stop/waiting
control-alt-delete stop/waiting
kexec-disable stop/waiting
quit-plymouth stop/waiting
rcS stop/waiting
prefdm stop/waiting
pref start/running, process 1507
init-system-dbus stop/waiting
splash-manager stop/waiting
start-ttys stop/waiting
rcS-sulogin stop/waiting
serial stop/waiting

# initctl status pref
pref start/running, process 1507

# initctl restart pref
pref start/running, process 2083

# initctl restart pref
initctl: Unknown instance:

# initctl restart pref
initctl: Unknown instance:

UPDATE2:

My script has 2 pecularities:

1) When it gets SIGTERM or SIGINT, it writes some data into PostgreSQL and this takes 10-15 seconds

2) When it is started numerous times, then the subsequent runs will fail immediately, because only the 1st instance will be able to listen at the TCP-port 8080

And in /var/log/messages I see:

...
17:44:25 static init: pref main process ended, respawning
17:44:26 static init: pref main process (2128) terminated with status 98
17:44:26 static init: pref main process ended, respawning
17:44:26 static init: pref main process (2133) terminated with status 98
17:44:26 static init: pref respawning too fast, stopped

is that all maybe the reason and is there something I could do? (maybe somehow delay the subsequent spawns?)

Solution 1

What does 'initctl list' show? Have you tried 'initctl reload-configuration' after creating the job?

Solution 2

The problem is that after your process seems to be terminating by itself. We don't have the log message for the process with pid 2083 but I suspect it died unexpectedly before you issued the next 'initctl restart pref' (like how pids 2128 and 2133 died for example). The key is that 'initctl restart foo' ONLY WORKS if the foo job name is still running. If it is dead, you need to do a normal 'initctl start foo'. I ran into this too. I was explicitly calling 'initctl stop foo' and then expecting 'initctl restart foo' to just work like they do with init scripts. They don't. You have to use 'initctl start foo'.

Comments

[sinping]

It looks like you can restart the service once going by your newly added output. Is that correct? What does status say after the first restart?

[sinping]

I see it in the man page for both RHEL and Ubuntu. Where did you get this information from?

[Alexander Farber]

I see "restart" in "man 5 init" for CentOS 6 :-)

[Simba Phoenix]

The command is 'initctl' not 'init'. If you view the man page at man 8 initctl, it only lists start, stop, and status as valid verbs for affecting the jobs. on the web, the man page is here [link]linux.die.net/man/8/initctl (linux.die.net/man/8/initctl)

[sinping]

It could be a vendor/distribution specific addition. In any event, the output above clearly shows the restart command working, which means it is there, which makes this answer and resulting conversation less than useful.

[sinping]

I am inclined to believe the delay is the cause. It is assigning a pid to the new process which dies and is no longer there. That makes more sense with the output you show above. I have found the debug.conf addition from this post very useful, you might try it and see if it gives you any additional insight. serverfault.com/questions/291546/centos-6-and-upstart

[Wesley]

FYI, there is no automatic way to include signatures in answers or questions on ServerFault because sigs are frowned upon. If any text does not obviously pertain to either the question or answers at hand, then it's likely to be discouraged. I removed your signature since it will likely be viewed by the community at large as being dangerously close to spamming for your company.

[S. Acarsoy]

You are a legend! Thank you. :) I needed to run initctl start instead of initctl restart. You would think they could give us a clearer error message.