How do I wait for nohup jobs to finish within a shell script?

Question

I have the following script:

echo "$wk" | while read -r a b;
do
    counter=$(($counter+1))
    nohup sh -c 'impala-shell -d nielsen -f query.sql --var=dt=$a --var=incre=$b;
    echo $a,$?>>$week_status_file;
    if [ $? -ne 0 ]; then
        echo $a,$b>>$err_week_file
    fi'
    if [ $(($counter%5)) == 0 -a $counter -ge 5 ];
    then
        echo "Check"
        wait
    fi
done

The idea is to kick off the wrapped command impala-shell... with two variables a and b (coming from the variable wk). This should be done in parallel with 5 processes. The next 5 jobs should wait until the previous 5 processes to finish. My current code prints out Check, but it doesn't wait at all. However, if I change wait to sleep 10s, it does sleep.

How do I make wait work here?

Solution

You have to supply process identifiers (PIDs) to wait. Also from your description it does not sound like your command is blocking which ultimately would make it harder to gather the PIDs you want to wait for. E.g. if the process is forking and allows you script to continue without knowing new forked processes' PID.

But in case your nohup sh -c ... line is blocking you want to send it to background with &, save the background processes' PID and finally wait for all the PIDs you saved (e.g. in $WAIT_FOR_PIDS). Variable $! gives you the PID of the last process send to background execution.

WAIT_FOR_PIDS="";
echo "$wk" | while read -r a b;
do
    counter=$(($counter+1))
    nohup sh -c 'impala-shell -d nielsen -f query.sql --var=dt=$a --var=incre=$b;
    echo $a,$?>>$week_status_file;
    if [ $? -ne 0 ]; then
        echo $a,$b>>$err_week_file
    fi' & WAIT_FOR_PIDS="$WAIT_FOR_PIDS $!"
    if [ $(($counter%5)) == 0 -a $counter -ge 5 ];
    then
        echo "Check"
        wait $WAIT_FOR_PIDS
        WAIT_FOR_PIDS=""
    fi
done

Also you should think about race conditions while echoing in parallel into $week_status_file. See man flock for easy shell accessible file locking.

Comments

[user1404316]

Just glancing at the code not in detail, you're using $? improperly. [ $? -ne 0 ] checks the exit code of the echo statement, not the nohup. Assign $? to something in order to use it more than once.

[lovechillcool]

@user1404316 just for your reference $? somehow refers the correct process. I tested using a failing scenario of impala-shell and $? passed 1 correctly.