How to read stderr from the process running in the background?

Question

I wanna send the daemon into the background and only continue my script execution when the daemon outputs the particular line to the stderr, something among the lines:

# Fictional daemon
{
    for x in {1..9}; do
        sleep 1
        if [ "$x" != "5" ]; then
            echo $x 1>&2
        else
            echo now 1>&2
        fi
    done
} &

# Daemon PID
PID="$!"

# Wait until the particular output... 
until { read -r line && grep -q "now" <<<"$line"; } do 
    sleep 1
done </proc/$PID/fd/2

#
# Do more stuff...
#

fg

Solution 1

With a mydaemon that behaves like:

#! /bin/sh -
i=1 stage=init
while true; do
  if [ "$i" -eq 5 ]; then
    echo >&2 ready
    stage=working
  fi
  echo >&2 "$stage $i"
  sleep 1
  i=$((i + 1))
done

That is, which takes 4 seconds to initialise, outputs ready when ready an carries on working from then all in the same process, you could write a start-mydaemon script like:

#! /bin/sh -
DAEMON=./mydaemon
LOGFILE=mydaemon.log
umask 027

: >> "$LOGFILE" # ensures it exists
pid=$(
  sh -c 'echo "$$"; exec "$0" "$@"' tail -fn0 -- "$LOGFILE" | {
    IFS= read -r tail_pid
    export LOGFILE DAEMON
    setsid -f sh -c '
      echo "$$"
      exec "$DAEMON" < /dev/null >> "$LOGFILE" 2>&1'
    grep -q ready
    kill -s PIPE "$tail_pid"
  }
)
printf '%s\n' "$DAEMON started in process $pid and now ready"

$ time ./start-mydaemon
./mydaemon started in process 230254 and now ready
./start-mydaemon  0.01s user 0.01s system 0% cpu 4.029 total
$ ps -fjC mydaemon
UID          PID    PPID    PGID     SID  C STIME TTY          TIME CMD
chazelas  230254    6175  230254  230254  0 10:28 ?        00:00:00 /bin/sh - ./mydaemon

start-mydaemon doesn't return until mydaemon has indicated that it was ready. Here, mydaemon's stdout and stderr goes to a logfile, while its stdin is redirected from /dev/null. setsid -f (not a standard command but found on most Linux distributions) should guarantee the daemon is detached from the terminal (if started from one).

Note however that if mydaemon fails to initialise and dies without ever writing ready, that script will wait forever for a ready that will never come (or will come the next time mydaemon is started successfully).

Also note that sh -c ...tail and the daemon are started simultaneously. If mydaemon has already initialised and printed ready by the time tail has started and seeked to the end of the log file, tail will miss the ready message.

You could address those problems with something like:

#! /bin/sh -
DAEMON=./mydaemon
LOGFILE=mydaemon.log
export DAEMON LOGFILE
umask 027

died=false ready=false canary_pid= tail_pid= daemon_pid=
: >> "$LOGFILE" # ensures it exists
{
  exec 3>&1
  {
    tail -c1 <&5 5<&- > /dev/null # skip to the end synchronously
    (
      sh -c 'echo "tail_pid=$$" >&3; exec tail -fn+1' |
        { grep -q ready && echo ready=true; }
    ) <&5 5<&- &
  } 5< "$LOGFILE"
  setsid -f sh -c '
    echo "daemon_pid=$$" >&3
    exec "$DAEMON" < /dev/null 3>&- 4>&1 >> "$LOGFILE" 2>&1' 4>&1 |
      (read anything; echo died=true) &
  echo "canary_pid=$!"
} | {
  while
    IFS= read -r line &&
      eval "$line" &&
      ! "$died" &&
      ! { [ -n "$daemon_pid" ] && "$ready" ; }
  do
    continue
  done
  if "$ready"; then
    printf '%s\n' "$DAEMON started in process $daemon_pid and now ready"
  else
    printf >&2 '%s\n' "$DAEMON failed to start"
  fi
  kill -s PIPE "$tail_pid" "$canary_pid" 2> /dev/null
  "$ready"
}

Though that's starting to be quite convoluted. Also note that since tail -f is now operating on stdin, on Linux, it will not use inotify to detect when new data is available in the file and resort to the usual check every second which means it may take up to one extra second to detect ready in the log file.

Solution 2

...
done </proc/$PID/fd/2

That doesn't work as you think it does.

The stderr of $PID is either

• the controlling tty, in which case you will try to read a string entered by the user at what it's probably also the stdin of $PID
• a pipe -- you would compete with whomever is already reading from it, resulting in a complete messup
• /dev/null -- EOF!
• something else ;-) ?

---

There are only hacky ways to redirect elsewhere a file descriptor from a running process, so your best bet is to have your input-waiting code downgrade itself into a cat >/dev/null running in the background.

For instance, this will "wait" until the daemon outputs 4:

% cat /tmp/daemon
#! /bin/sh
while sleep 1; do echo $((i=i+1)) >&2; done

% (/tmp/daemon 2>&1 &) | (sed /4/q; cat >/dev/null &)
1
2
3
4
%

After which /tmp/daemon will continue writing to cat >/dev/null &, outside the control of shell.

Another solution would be to redirect the stderr of the daemon to some regular file and tail -f on it, but then the daemon will go on filling your disk with garbage (even if you rm the file, the space it occupies won't be freed until the daemon closes it), which is even worse than having a low-resource cat loitering around.

Of course, the best thing will be to write /tmp/daemon as a real daemon, which backgrounds itself after initializing, closes its std file descriptors, uses syslog(3) for printing errors, etc.

Comments

[Uncle Billy]

when the code reading from the fifo has stopped, the daemon will continue to try writing into it until it fills up and the daemon will block until something is going to read from the fifo.

[Ali Hassan]

for loop is here just to illustrate what I'm trying to achieve. In fact I have RabbitMQ binary, which I need to start first and then initialize the internal state of it by connecting to it using cli app. If I send it to the background my init scripts starts running too early, when the server is not ready yet to accept connections. I know I could just sleep for a certain number of seconds, but I wanna be a bit smarter :-)

[Uncle Billy]

So it's my /tmp/daemon. Just for illustration. I don't see what difference that makes. The sed /now/q is just a nicer way for writing your while ... read ... grep -q loop.

[Ben]

What is fd 4 used for and why are you redirecting it?