Waiting for background processes to finish before exiting script

Question

How do I make sure that all my background processes have finished execution before I exit my script (TCL/Bash).

I was thinking of writing all my background process pids to a pidfile. And then at the end pgrep the pidfile to see if any processes are still running before I exit.

Is there some simpler way to do this? And is there a TCL specific way to do this?

Solution 1

If you want to wait for jobs to finish, use wait. This will make the shell wait until all background jobs complete. However, if any of your jobs daemonize themselves, they are no longer children of the shell and wait will have no effect (as far as the shell is concerned, the child is already done. Indeed, when a process daemonizes itself, it does so by terminating and spawning a new process that inherits its role).

#!/bin/sh
{ sleep 5; echo waking up after 5 seconds; } &
{ sleep 1; echo waking up after 1 second; } &
wait
echo all jobs are done!

Solution 2

You can use kill -0 for checking whether a particular pid is running or not.

Assuming, you have list of pid numbers in a file called pid in pwd

while true;
do 
    if [ -s pid ] ; then
        for pid in `cat pid`
        do  
            echo "Checking the $pid"
            kill -0 "$pid" 2>/dev/null || sed -i "/^$pid$/d" pid
        done
    else
        echo "All your process completed" ## Do what you want here... here all your pids are in finished stated
        break
    fi
done

Solution 3

WARNING: Long script ahead.

A while ago, I faced a similar problem: from a Tcl script, launch a number of processes, then wait for all of them to finish. Here is a demo script I wrote to solve this problem.

main.tcl

#!/usr/bin/env tclsh

# Launches many processes and wait for them to finish.
# This script will works on systems that has the ps command such as
# BSD, Linux, and OS X

package require Tclx; # For process-management utilities

proc updatePidList {stat} {
    global pidList
    global allFinished

    # Parse the process ID of the just-finished process
    lassign $stat processId howProcessEnded exitCode

    # Remove this process ID from the list of process IDs
    set pidList [lindex [intersect3 $pidList $processId] 0]
    set processCount [llength $pidList]

    # Occasionally, a child process quits but the signal was lost. This
    # block of code will go through the list of remaining process IDs
    # and remove those that has finished
    set updatedPidList {}
    foreach pid $pidList {
        if {![catch {exec ps $pid} errmsg]} {
            lappend updatedPidList $pid
        }
    }

    set pidList $updatedPidList

    # Show the remaining processes
    if {$processCount > 0} {
        puts "Waiting for [llength $pidList] processes"
    } else {
        set allFinished 1
        puts "All finished"
    }
}

# A signal handler that gets called when a child process finished.
# This handler needs to exit quickly, so it delegates the real works to
# the proc updatePidList
proc childTerminated {} {
    # Restart the handler
    signal -restart trap SIGCHLD childTerminated

    # Update the list of process IDs
    while {![catch {wait -nohang} stat] && $stat ne {}} {
        after idle [list updatePidList $stat]
    }
}

#
# Main starts here
#

puts "Main begins"
set NUMBER_OF_PROCESSES_TO_LAUNCH 10
set pidList {}
set allFinished 0

# When a child process exits, call proc childTerminated
signal -restart trap SIGCHLD childTerminated

# Spawn many processes
for {set i 0} {$i < $NUMBER_OF_PROCESSES_TO_LAUNCH} {incr i} {
    set childId [exec tclsh child.tcl $i &]
    puts "child #$i, pid=$childId"
    lappend pidList $childId
    after 1000
}

# Do some processing
puts "list of processes: $pidList"
puts "Waiting for child processes to finish"
# Do some more processing if required

# After all done, wait for all to finish before exiting
vwait allFinished

puts "Main ends"

child.tcl

#!/usr/bin/env tclsh
# child script: simulate some lengthy operations

proc randomInteger {min max} {
    return [expr int(rand() * ($max - $min + 1) * 1000 + $min)]
}

set duration [randomInteger 10 30]
puts "  child #$argv runs for $duration miliseconds"
after $duration
puts "  child #$argv ends"

Sample output for running main.tcl

Main begins
child #0, pid=64525
  child #0 runs for 17466 miliseconds
child #1, pid=64526
  child #1 runs for 14181 miliseconds
child #2, pid=64527
  child #2 runs for 10856 miliseconds
child #3, pid=64528
  child #3 runs for 7464 miliseconds
child #4, pid=64529
  child #4 runs for 4034 miliseconds
child #5, pid=64531
  child #5 runs for 1068 miliseconds
child #6, pid=64532
  child #6 runs for 18571 miliseconds
  child #5 ends
child #7, pid=64534
  child #7 runs for 15374 miliseconds
child #8, pid=64535
  child #8 runs for 11996 miliseconds
  child #4 ends
child #9, pid=64536
  child #9 runs for 8694 miliseconds
list of processes: 64525 64526 64527 64528 64529 64531 64532 64534 64535 64536
Waiting for child processes to finish
Waiting for 8 processes
Waiting for 8 processes
  child #3 ends
Waiting for 7 processes
  child #2 ends
Waiting for 6 processes
  child #1 ends
Waiting for 5 processes
  child #0 ends
Waiting for 4 processes
  child #9 ends
Waiting for 3 processes
  child #8 ends
Waiting for 2 processes
  child #7 ends
Waiting for 1 processes
  child #6 ends
All finished
Main ends

Solution 4

GNU parallel and xargs

These two tools that can make scripts simpler, and also control the maximum number of threads (thread pool). E.g.:

seq 10 | xargs -P4 -I'{}' echo '{}'

or:

seq 10 | parallel -j4  echo '{}'

See also: how to write a process-pool bash shell

View more solutions

Comments

[egorulz]

I tried this out but got the following error: can't wait for variable "allFinished": would wait forever while executing "vwait allFinished"

[Hai Vu]

I tested this solution on my Mac, but not on other platforms. If I have time, I'll test it on Linux. However, I don't have a Windows machine to find out. Are you running it on Windows?

[egorulz]

Actually I'm running this on a Freebsd machine.

[Hai Vu]

I have tested on Linux and it works. I don't understand why it does not work on FreeBSD.

[egorulz]

Are there possibilities for timing issues? In my test I had 3 background processes with the last one running for around 20s and it was on that run that I got the error.

[Hai Vu]

I borrowed a PC-BSD 9.1 virtual machine, tried it out and found no problem. I wonder if we can get around by creating a loop to wait for the allFinished variable, instead of using the vwait command. That seems to be re-inventing the wheels, though.

[Hai Vu]

let us continue this discussion in chat

[Yordan Georgiev]

+1 I would add only the explanation of how-to store the child procs: cat $items_list_file | { #spone kids ( some_command )& echo $! >> pid }

[smbear]

In general, you can't be sure that a PID you're testing still identifies your process. Due to PID recycling, it may happen that some other process will get the PID which you were testing. More about PID recycling in Linux: goo.gl/eZceq2

[Marco Marsala]

It works but I'm getting the warning : command not found while running this simple script: sleep 10 & wait Btw, apart the annoying warning, the wait command works as intended. Any idea about what is causing such warning? - NB: there is a carriage return between sleep 10 & and wait but the forum is stripping them

[guest]

@MarcoMarsala the & is followed by the next command, and the interpreter will try to look up a program whose name is a carriage return.