Upstart process listed as stop/waiting

Question

I have this upstart script

When I run: sudo start poxa it starts the process but when I run: initctl list it shows me that is is stop/waiting but when I check ps aux | grep poxa it show the process.

And, as expected, when I try to stop it: sudo stop poxa it returns: stop: unknown instance:

# Upstart Configuration
# put on /etc/init
description     "Poxa"
author          "Poxa"

start on (filesystem or runlevel [2345])
stop on runlevel [!2345]

#respawn
#respawn limit 10 5
umask 022

console none

pre-start script
    test -x /home/ec2-user/poxa/rel/poxa/bin/poxa || { stop; exit 0; }
end script

pre-stop script
        echo "[`date -u +%Y-%m-%dT%T.%3NZ`] Stoping..." >> /var/log/poxa.log 2>&1
end script

script
        export HOME=/home/ec2-user
        echo "[`date -u +%Y-%m-%dT%T.%3NZ`] Starting..." >> /var/log/poxa.log 2>&1
        exec /home/ec2-user/poxa/rel/poxa/bin/poxa start >> /var/log/poxa.log 2>&1
end script

ps aux | grep poxa

root     29032  0.0  0.1  13656  1720 ?        S    11:42   0:00 /usr/local/lib/erlang/erts-7.0/bin/run_erl -daemon /home/ec2-user/poxa/rel/poxa/tmp/erl_pipes/poxa/ /home/ec2-user/poxa/rel/poxa/log exec "/home/ec2-user/poxa/rel/poxa/bin/poxa" "console"
root     29033  0.3  2.0 318992 21100 pts/1    Ssl+ 11:42   0:00 /usr/local/lib/erlang/erts-7.0/bin/beam -- -root /home/ec2-user/poxa/rel/poxa -progname home/ec2-user/poxa/rel/poxa/releases/0.4.3/poxa.sh -- -home /home/ec2-user -- -boot /home/ec2-user/poxa/rel/poxa/releases/0.4.3/poxa -boot_var ERTS_LIB_DIR /usr/local/lib/erlang/erts-7.0/../lib -config /home/ec2-user/poxa/rel/poxa/running-config/sys.config -pa /home/ec2-user/poxa/rel/poxa/lib/consolidated -name [email protected] -setcookie
poxa -user Elixir.IEx.CLI -extra --no-halt +iex -- console

I have made this: http://upstart.ubuntu.com/cookbook/#how-to-establish-fork-count to get the fork count but it returns 44!

Solution

Upstart daemon works with three daemon mode: no expect, expect fork, expect daemon, 6.13.5 Implications of Misspecifying expect

As you can see at there, you need to check the process fork count.

If you insert expect fork, the "exec /home/ec2-user/poxa/rel/poxa/bin/poxa start" process needs to fork once and exit. Also it needs to fork twice and exit for expect daemon. The upstart will trace that last child pid.

To run the daemon correctly with the upstart script, you need to check the fork of poxa process.

Following is my example for upstart and simple daemon.

upstart script in /etc/init

# egservice - eg daemon
#
# This is an example
# upstart script.

description    "egservice"

start on runlevel [2345]
stop on runlevel [!2345]
#LOOK Following line, So egdaemon needs to fork just once.
expect fork
respawn

exec /sbin/egdaemon

egdaemon.c for /sbin/egdaemon

#include <stdio.h>

int main()
{
    /* LOOK following line, just once fork and main does not wait child and die, 
    so new child will be changed to a init's child and also a daemon. */
    int pid = fork();
    if (pid == 0) {
        while (1) {
            printf("Example daemon\n");
            sleep(1);
        }
    }
    return 0;
}

And this C source file can be compiled to a executable binary with following command.

$ gcc -o egdaemon egdaemon.c

Comments

[PKumar]

what does syslog have regarding this process ?

[Admin]

@PKumar I don't have it installed but with ps aux | grep poxa I can see the process running

[PKumar]

then upstart is unable to track running pid check log file while starting and stoping the service

[Admin]

@PKumar yes, that's exactly the problem and I have no idea why upstart is not tracking it. The log is generated correctly when I run sudo start poxa but since it doesn't track the PID I can't stop it, upstart is considering it as a stopped job. Maybe forcing a PID?

[Admin]

@PKumar I have added the output when checking for the poxa process, as you can see it has two processes running

[Admin]

@PKumar I had to add the expect fork command to my script ;)

[Admin]

@PKumar but now it doesn't stop hahah it looks like it is processing something but it doesn't stop

[Admin]

@PKumar I was wrong, using expect fork didn't fix the problem completely, it started the process and I could check that it was running with initctl list but I can't stop it, it hangs and initctl list shows a different PID number than the one showed with ps aux

[user.dz]

@Gerep, Could you try expect daemon , ref askubuntu.com/questions/89518/…

[Marcos Silveira]

Didi you tried start on runlevel [2345] instead start on (filesystem or runlevel [2345])???

[xiaodongjie]

What is the output of the service poxa start? ans about "ps ax | grep poxa" ?