Testing an infinite loop in the background within Docker?

Question

I want to run a background queue working in my docker image:

php artisan queue:work --daemon --sleep=1 --tries=3 &

Immediately after that it starts Apache httpd running a PHP laravel app. The laravel app sends push notifications out to Redis. The background queue worker collects the messages from Redis and sends them via a push service.

That works and logs out every second. In case the background command crashes I would like it to restart. According to this answer https://unix.stackexchange.com/a/223780/72040 I can run a command infinitely in the background with something like:

while true; do php $QUEUE_WORK; done &

To test this I ran my container with docker run then did docker exec -it to login and see:

UID        PID  PPID  C STIME TTY          TIME CMD
100000       1     0  0 19:55 ?        00:00:00 httpd -D FOREGROUND
100000      21     1  0 19:55 ?        00:00:00 /bin/sh -e /tmp/scripts/run
100000      22    21  1 19:55 ?        00:00:01 php artisan queue:work --sleep=1 --tries=3
100000      43     1  0 19:55 ?        00:00:00 /usr/bin/cat
100000      50     1  0 19:55 ?        00:00:00 /usr/bin/cat
100000      51     1  0 19:55 ?        00:00:00 /usr/bin/cat
100000      52     1  0 19:55 ?        00:00:00 /usr/bin/cat
100000      53     1  0 19:55 ?        00:00:00 httpd -D FOREGROUND
...
100000     130     0  1 19:57 pts/0    00:00:00 /bin/bash
100000     144   130  0 19:57 pts/0    00:00:00 ps -efww

I then run kill 22 see output on the docker run of:

/tmp/scripts/run: line 10:    22 Killed                  php $QUEUE_WORK

The loop doesn't keep the process up. I have double checked that the loop is the code being run in the image. If I have the loop run a ”echo x: sleep 1” it runs okay.

Why doesn't the loop replace the command when I kill it?

NOTE: The application is deployed as multiple pods on Kubernetes with monitoring and automated restarts if the httpd health check URL returns an error or time-out.

I don't want to deploy the queue workers as a separate pod or as a sidecar container I want to run it as a background process in the httpd container so that it shares the application code with httpd with zero config.

I am not interested in running process monitors or other tools or technology to ”keep alive”.

My question is why does the Bash loop exit on kill of the process it is running?

Solution

Setup

I set up the following Dockerfile:

$ more Dockerfile
From centos
ADD run.sh /tmp/run.sh
RUN chmod +x /tmp/run.sh
ENTRYPOINT ["/tmp/run.sh"]

Setup a script, run.sh:

$ cat run.sh
while true; do sleep 15 ; echo "background"; done &

while true; do sleep 12 ; echo "foreground"; done

Built it:

$ docker build -t sleeper .
Sending build context to Docker daemon 7.791 MB
Step 1/4 : FROM centos
 ---> 49f7960eb7e4
Step 2/4 : ADD run.sh /tmp/run.sh
 ---> b4099de53780
Removing intermediate container 1ce8e3a1dac5
Step 3/4 : RUN chmod +x /tmp/run.sh
 ---> Running in e410429a6cba

 ---> 06789467e636
Removing intermediate container e410429a6cba
Step 4/4 : ENTRYPOINT /tmp/run.sh
 ---> Running in ad8b847b505f
 ---> a2415df63f99
Removing intermediate container ad8b847b505f
Successfully built a2415df63f99

Example

Then started it up:

$ docker run -dit sleeper
28c19c338e6e6177529cf989f42c7f14b17f1c705a61f5244d5350f0ab8f8364

Then it runs repeatedly showing:

foreground 
background 
foreground 
background

Now when I watch it I can see that it's staying up, in spite of the sleep commands eventually "dying":

$ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED              STATUS              PORTS               NAMES
28c19c338e6e        sleeper             "/tmp/run.sh"       About a minute ago   Up About a minute                       focused_lumiere

From above we can see that this container has been up in excess of 1 minute. Here's what the ps auxf looks like of the inside of the container:

$ ps auxf
...
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root        24  0.0  0.1  11828  2964 pts/1    Ss   16:00   0:00 /bin/bash
root        58  0.0  0.1  51712  3432 pts/1    R+   16:01   0:00  \_ ps auxf
root         1  0.0  0.1  11692  2572 pts/0    Ss+  15:58   0:00 /bin/bash /tmp/
root         5  0.0  0.1  11696  2272 pts/0    S+   15:58   0:00 /bin/bash /tmp/
root        56  0.0  0.0   4368   636 pts/0    S+   16:01   0:00  \_ sleep 15
root        57  0.0  0.0   4368   664 pts/0    S+   16:01   0:00 sleep 12

Now if we kill the background sleep 15 like so:

$ kill 56

And run another docker ps:

$ docker ps
...
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root        24  0.0  0.1  11828  2964 pts/1    Ss   16:00   0:00 /bin/bash
root        60  0.0  0.1  51712  3344 pts/1    R+   16:01   0:00  \_ ps auxf
root         1  0.0  0.1  11692  2572 pts/0    Ss+  15:58   0:00 /bin/bash /tmp/
root         5  0.0  0.1  11696  2272 pts/0    S+   15:58   0:00 /bin/bash /tmp/
root        59  0.0  0.0   4368   636 pts/0    S+   16:01   0:00  \_ sleep 15
root        57  0.0  0.0   4368   664 pts/0    S+   16:01   0:00 sleep 12

We can see that the while loop that's guarding our background sleep 15 process has done its job and restarted another sleep 15.

Comments

[Jesse Chisholm]

re: why didn't the while catch it? the while loop itself does not put the php command in a subshell, so the pid for the php command is effectively the pid for the while command as well. You could address this by forcing the php command into a subshell with while true; do (php args ...); done.

[simbo1905]

i have accepted your answer as it shows my code will work in prod if the queue worker decides to die a natural death, i don't now why the php behaves differently. yet i have run out of curiosity for the time being. thanks.