Why isn't my upstart service starting on system boot?

Question

Following on from this question, I have written a simple upstart service (/etc/init/pms.conf) for my headless Ubuntu Server 11.04 box as follows:

start on filesystem and net-device-up IFACE=eth0
stop on runlevel [016]
respawn

exec /home/administrator/pms-current/PMS.sh

I can start (or stop) this service at will from the command line:

service pms start

And I can see that it is indeed running.

However, when I first boot my machine the service does not start. If I SSH into the box and check the service status I get:

$ service pms status
pms stop/waiting

My question is why is this happening? Why isn't my service starting on boot?

UPDATE 1: unsure whether my service was being started and subsequently dying or just wasn't being start at all, I added the following to PMS.sh:

echo "STARTED" > $STARTLOG

This obviously just gives me something to look for. I tested this by starting the service myself and then checking start.log. I then deleted the start.log and rebooted. It wasn't there after the restart, so it seems as though upstart definitely isn't starting my service. I suppose it could be dying at an earlier point in the process, but that seems rather unlikely given the simplicity of it all.

UPDATE 2: I've just upgraded to 11.10 which includes an upstart upgrade, but this problem still occurs.

UPDATE 3: As requested, I've booted with --debug. The output of cat /var/log/syslog | grep init is too long to place in the question, but you view it here.

UPDATE 4: More logs, this time the upstart conf is included at the top. Run 1 and run 2.

Solution 1

I would recommend increasing the verbosity of the job, e.g. by using pre-start/post-start entries.

pre-start script
  logger "pre-start for myprog"
end script

post-start script
  logger "post-start for myprog"
end script

# and for PMS itself:
script
  logger "just before executing PMS"
  exec /home/administrator/pms-current/PMS.sh
end script

Further information at http://upstart.ubuntu.com/cookbook/

Also have a look at http://upstart.ubuntu.com/wiki/Debugging

Solution 2

What's probably happening here is that pms is starting before your network adapters come up, and probably before even the loopback adapter (lo). Assuming we're talking about PS3 Media Server, it's a networked service and it probably doesn't like starting up with no interfaces available.

Try changing your start on criteria to:

start on filesystem and net-device-up IFACE!=lo

Meaning, start after any "real" network interface is up. However, that might not be ideal, if eth0 is the next interface up, PMS starts, but you really want PMS to use wlan0, that won't do. The service will start but it might not have been able to pick the interface you wanted it to listen on. Assuming you know the interface you're going to stream over and it won't be changing, I would hardcode it into the job, e.g.:

start on filesystem and net-device-up IFACE=wlan0

On Oneiric (11.10), you can use the event static-network-up to wait for all statically configured devices. Which is nice because it allows you to write network-dependent jobs without hardcoding an interface. [Note: by "all statically configured devices", I'm referring to using /etc/network/interfaces instead of NetworkManager. It does not mean static in the sense of static IP vs. DHCP.]

Solution 3

Managed to fix similar problem by using start on runlevel instead:

start on runlevel [2345]

Solution 4

From examining your syslog the pms process starts with no errors but then after a short while its goal is changed from start to stop meaning it is killed.

This is slightly strange because you have added the repsawn clause so it should attempt to start again after it is stopped but it never does. So I'm guessing you removed the respawn clause.

Between the pms service starting and stopping only 2 services are started ufw and network-interface (eth0), and 1 is started udev-fallback-graphics.

It seems that you process pms is being started in parallel. Unfortunately the upstart documentation is a little bit hazy on the exact differences between start on ... vanilla and start on starting ... and start on started ....

Try changing your startup stanza to

start on started networking

or just too

start on net-device-up IFACE=eth0

The log output is slightly strange as the net-device-up event comes much later but pms starts before it.

This should ensure that your process only starts once all networking set up is finished i.e. the job has not only started but finished.

Also do not trust log output completely, early in the boot process logging output to any file does not always work. See the answer in Debugging Upstart

Solution 5

I had the same problem and eventually I solved it simply with:

start on runlevel [2345]

without any net-device-up or started networking stuff

This is the complete upstart script, and it works perfectly:

# MyApp

description     "MyApp"
author          "me"

start on runlevel [2345]
stop on runlevel [016]

respawn

exec /usr/bin/myapp 2>> /var/logs/myapp.log

View more solutions

Comments

[Kent Boogaart]

This sounded like the trick, but it didn't work. I only have lo and eth0 but I used your second suggestion: start on filesystem and net-device-up IFACE=eth0. Still no go after a reboot. I've just noticed something in the PMS log that may be a lead. I'll investigate and get back...

[Mark Russell]

It's interesting. One thing I didn't mention is that I tried your original script and it worked on boot for my machine. I attributed that to just the luck of the draw (i.e. in my race condition, the good car won, and in yours the bad car won). I really can't see what other dependency we're missing here. Weird.

[Mark Russell]

Since you can start it after booting up, we must be missing another service dependency. One dirty hack that might work (but won't illuminate us at all) is to just drop in a sleep 10 -- or higher -- in a "pre-start script" before exec'ing the shell script.

[Kent Boogaart]

Sorry Mark - but we're on the same page. I tried the sleep 10 thing already in a pre-start script. No go. Then I tried deleting the debug.log altogether and rebooting. After boot I had the same service status, and no debug.log file, so I'm not convinced PMS is actually run at all. Is there an easy way to diagnose this? If I alter the PMS.sh to spit out some output, where will it go? I suppose I could always direct it to my own file - might give that a shot next.

[Kent Boogaart]

I just updated my question with more info.

[Mark Russell]

To see if Upstart is even trying, I'd go with --verbose on the kernel command line. upstart.ubuntu.com/cookbook/…

[Kent Boogaart]

Thanks - I'll give that a try tomorrow. Might watch a movie via PMS tonight! ;)

[Kent Boogaart]

Just tried it. I get nothing but a blank screen if I specify --debug :S

[Kent Boogaart]

This really is doing my head in. I've tried a dozen different things off the back of your post. All have failed with varying obscure messages in the logs. My latest attempt resulted in init: pms main process (1329) terminated with status 143, which means naught to me. I can see that PMS.sh isn't even being started because the first thing it does is write to its own log, and that log entry isn't present. I can see my pre-start output, which tells me that the target file exists and is executable. I will pick this up again tomorrow, but if you have any ideas I'd love to hear them. Thanks.

[Daniele B]

hi @KentBoogaart, I seem to have your same problem. Have you found a solution?

[Phương Nguyễn]

I see people keep saying "add --verbose on the kernel command line". How would I do that?

[Mevin Babu]

@KentBoogaart I have the same problem as you? Any luck with yours ?

[slm]

The link referenced is broken.