Upstart: stop on runlevel [016] vs stop on starting rc RUNLEVEL=[016]

Question

So, I'm working on an upstart init script for mysql. Should be trivial, right? My stop stanza looks like:

kill timeout 30
stop on runlevel [016]

I shutdown now -r, and mysql yells about my MyISAM tables being crashed, just like I'd kill -9'd my mysql process. Boo. Upstart doesn't seem to be waiting for mysql to quit before restarting. Some searching on Google finds me this link suggesting I do, instead:

stop on starting rc RUNLEVEL=[016]

and indeed, that seems to fix the problem.

But: WFT? Essentially, I want to make sure mysql is stopped before trying to unmount the local filesystems. Is this the right way to do this? Why is my stop on runlevel stanza not waiting for mysql to stop before switching runlevels?

And: is this y'know, documented anywhere?

Solution

The issue he is actually caused by Upstarts flexibility in handling both Upstart jobs and SystemV services on Ubuntu.

For information on events, a good place to look is the man pages on your system. If you're running Ubuntu Natty or newer, you'll now have upstart-events(7):

man 7 upstart-events

This gives you a lot of information. Here's an online version of (most of ) that man page:

http://upstart.ubuntu.com/cookbook/#ubuntu-well-known-events-ubuntu-specific

Currently, the shutdown for an Ubuntu system is actually finalized by the SysV side of the work (for historical reasons). If you 'stop on runlevel [016]', your job will start to stop when that runlevel event is emitted. However, if it takes too long, the SystemV part of the system (/etc/init.d/*) will actually take control and shut the system down. Whereas, if you 'stop on starting rc RUNLEVEL=[016]', Upstart (not SystemV) will run your job and once mysql has stopped, then continue to start the SystemV shutdown sequence.

The Upstart Cookbook contains lots of examples like this:

• http://upstart.ubuntu.com/cookbook/
• http://upstart.ubuntu.com/cookbook/upstart_cookbook.pdf