How can I make puppet stop a service before replacing a file?

Question

I'm trying to avoid a race condition when replacing the software behind a puppet Service.

To do that, puppet needs to stop the service, replace the executable, then start the service. Is there a way to talk puppet into doing that? Its preferred way of doing things seems to be to replace the executable, then check the status and start the service again if necessary.

(This example is contrived. The real race condition is nowhere near this simple...)

Here's the puppet manifest I'm using to simulate this problem:

$O = '1'
$I = '2'

exec { hi :
        command => '/bin/echo "$(/bin/date +%s) puppet says hello" >> /tmp/freebird.log' ,
        }

file { exe :
        name => "/tmp/freebird" ,
        ensure => present ,
        owner => "root" ,
        group => "root" ,
        mode => "0555" ,
        source => "/root/test-v$I" ,
        }

file { init :
        name => '/etc/init.d/freebird' ,
        ensure => present,
        owner => "root",
        group => "root",
        mode => "0555",
        source => "/root/test.init" ,
        }

service { freebird :
        ensure => running,
        enable => true,
        hasrestart => true,
        hasstatus => true,
        require => [ File[init], File[exe] ],
        }

Here's the test-v1 file. The test-v2 file is the same but with v=2.

#!/bin/bash
v=1

while true
do
        echo "$(date +%s) $v" >> /tmp/freebird-v.log
        sleep 1
done

And the init.d script:

#!/bin/bash
#
# /etc/rc.d/init.d/freebird

# chkconfig: 2345 90 10
# description:       freebird
# Provides:          freebird
# Required-Start:    $syslog $remote_fs
# Should-Start:
# Required-Stop:     $syslog $remote_fs
# Should-Stop:
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description:  freebird 


# Source function library.
. /etc/rc.d/init.d/functions

xme=freebird
export PATH=/sbin:/bin:/usr/sbin:/usr/bin

function L () {
        echo "$(date +%s) $*" 1>&2
        echo "$(date +%s) $*" >> /tmp/$xme.log
        }


case "$1" in
        (start) L $1 $xme
                ( /tmp/$xme &)
                ;;
        (stop) L $1 $xme
                fuser -k /tmp/$xme
                ;;
        (status) L $1 $xme
                /sbin/fuser /tmp/$xme >/dev/null 2>&1
                ;;
        (restart) L $1 $xme
                $0 stop
                $0 start
                ;;
        (*)
                echo "Usage: $xme {start|stop|status|restart]"
                exit 1
                ;;
esac

Solution 1

I'm trying to avoid a race condition when replacing the software behind a puppet Service.

To do that, puppet needs to stop the service, replace the executable, then start the service. Is there a way to talk puppet into doing that? Its preferred way of doing things seems to be to replace the executable, then check the status and start the service again if necessary.

So the problem with what Puppet is currently doing is that it should always be restarting the service after replacing certain files?

If so, you should use a notify/subscribe relationship to always trigger the service restart after files have been replaced. Taking your example service, we can add subscriptions onto the files that make it up (in the same way you might with a config) and this will trigger a restart if either of them changes.

service { freebird :
        ensure => running,
        enable => true,
        hasrestart => true,
        hasstatus => true,
        require => [ File[init], File[exe] ],
        subscribe => [ File[init], File[exe] ],
        }

The other way of doing it is to use your OS package management, which Puppet has good support for. You would then trigger the restart (or a stop/start in pre/post install) from the package scripts, leaving Puppet to ensure the service is configured and running. Have a look at Jordan Sissel's fpm project for a tool to easily build many package formats.

Solution 2

I will rephrase the question for clarity and search, then provide a solution.

I would suggest however, that if you have the option to do so, use the pre-install feature of your native packaging system.

Q: How to emulate rpm's pre-install script via puppet. One use case is to stop the puppet service before installing the executable, then start it up again after replacing the file. This is in contrast to puppet's normal ordering of replace the file, then restart the service.

Fortunately, my use case already requires the symlink mess. If yours doesn't, please post your solution.

To run the test comprised of the files below, I edit $tversion in test.pp then paste this into my terminal:

fuser /tmp/freebird-v.log /tmp/freebird
: > /tmp/freebird.log
echo ==== >> /tmp/freebird.log ; puppet apply --verbose --onetime --no-daemonize test.pp 2>&1 | tee ~/D ; cat /tmp/freebird.log
ps auxww|grep freebird
fuser /tmp/freebird-v.log /tmp/freebird

File test.pp:

$tversion = '1'

exec { hi :
        command => '/bin/echo "$(/bin/date +%s) puppet says hello" >> /tmp/freebird.log' ,
        }

file { exe :
        name => "/tmp/freebird-v$tversion" ,
        ensure => present ,
        owner => "root" ,
        group => "root" ,
        mode => "0555" ,
        content => template("/root/test-template") ,
        }

file { exe_ln :
        name => "/tmp/freebird" ,
        ensure => link ,
        owner => "root" ,
        group => "root" ,
        mode => "0555" ,
        target => "/tmp/freebird-v$tversion" ,
        }

file { init :
        name => '/etc/init.d/freebird' ,
        ensure => present,
        owner => "root",
        group => "root",
        mode => "0555",
        source => "/root/test.init" ,
        }

exec { freebird_stop_if_incoherent :
        command => '/sbin/service freebird stop' ,
        refreshonly => false , # required for when entering at exe_ln
        onlyif => "/sbin/service freebird status && ! test /tmp/freebird -ef '/tmp/freebird-v$tversion'" , # short-circuits the refreshonly for most cases
        }

service { freebird :
        ensure => running,
        enable => true,
        hasrestart => true,
        hasstatus => true,
        }

File[exe_ln]            <~ Exec[freebird_stop_if_incoherent]
Service[freebird]       <- File[exe_ln]

File test-template:

#!/bin/bash
v=<%= tversion %>

while true
do
        echo "$(date +%s) $v" >> /tmp/freebird-v.log
        sleep 1
done

File test.init:

#!/bin/bash
#
# /etc/rc.d/init.d/freebird

# chkconfig: 2345 90 10
# description:       freebird
# Provides:          freebird
# Required-Start:    $syslog $remote_fs
# Should-Start:
# Required-Stop:     $syslog $remote_fs
# Should-Stop:
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description:  freebird 


# Source function library.
. /etc/rc.d/init.d/functions

xme=freebird
export PATH=/sbin:/bin:/usr/sbin:/usr/bin

function L () {
        local pid=$$
        local ppid=$(ps l $pid |awk '{print $4}' |tail -1)
        local extra="-- $(ps $ppid|tail -1|sed 's,^[^/]*/,/, ; s,/[0-9][^/]*/,/,')"
        echo "$(date +%s) $pid $ppid $* $extra" 1>&2
        echo "$(date +%s) $pid $ppid $* $extra" >>/tmp/$xme.log 2>&1
        }


case "$1" in
        (start) L $1 $xme
                fuser /tmp/$xme >/dev/null 2>&1 || ( /tmp/$xme &)
                ;;
        (stop) L $1 $xme
                fuser /tmp/$xme 2>&1
                fuser -k /tmp/$xme 1>&2 ||true
                fuser /tmp/$xme 2>&1
                true
                ;;
        (status) L $1 $xme
                /sbin/fuser /tmp/$xme >/dev/null 2>&1
                ;;
        (restart) L $1 $xme
                fuser -k /tmp/$xme 1>&2 ||true
                ( /tmp/$xme &)
                ;;
        (*)
                echo "Usage: $xme {start|stop|status|restart]"
                exit 1
                ;;
esac

Comments

[bugi]

M0dlx, thank you. Your comment pushed me onto the right path. If I were to do this all from scratch, I would definitely use the OS packager's pre-install feature instead of forcing puppet to handle this. And fpm makes that much easier...

[harshad]

@m0dlx, I have one doubt. What if my manifest is like File['X'],File['Y'],File['Z']->Service['XYZ']{subscribe: File['X','Y']}? Will the service be executed/restarted only after file 'Z' has been created or it'll restart @ every moment file X/Y/Z has been modified.