How can I make Perl wait for child processes started in the background with system()?

Question

I have some Perl code that executes a shell script for multiple parameters, to simplify, I'll just assume that I have code that looks like this:

for $p (@a){
    system("/path/to/file.sh $p&");
}

I'd like to do some more things after that, but I can't find a way to wait for all the child processes to finish before continuing.

Converting the code to use fork() would be difficult. Isn't there an easier way?

Solution 1

Using fork/exec/wait isn't so bad:

my @a = (1, 2, 3);
for my $p (@a) {
   my $pid = fork();
   if ($pid == -1) {
       die;
   } elsif ($pid == 0) {
      exec '/bin/sleep', $p or die;
   }
}
while (wait() != -1) {}
print "Done\n";

Solution 2

You are going to have to change something, changing the code to use fork is probably simpler, but if you are dead set against using fork, you could use a wrapper shell script that touches a file when it is done and then have your Perl code check for the existence of the files.

Here is the wrapper:

#!/bin/bash

$*

touch /tmp/$2.$PPID

Your Perl code would look like:

for my $p (@a){
    system("/path/to/wrapper.sh /path/to/file.sh $p &");
}
while (@a) {
    delete $a[0] if -f "/tmp/$a[0].$$";
}

But I think the forking code is safer and clearer:

my @pids;
for my $p (@a) {
    die "could not fork" unless defined(my $pid = fork);\
    unless ($pid) { #child execs
        exec "/path/to/file.sh", $p;
        die "exec of file.sh failed";
    }
    push @pids, $pid; #parent stores children's pids
}

#wait for all children to finish
for my $pid (@pids) {
    waitpid $pid, 0;
}

Solution 3

Converting to fork() might be difficult, but it is the correct tool. system() is a blocking call; you're getting the non-blocking behavior by executing a shell and telling it to run your scripts in the background. That means that Perl has no idea what the PIDs of the children might be, which means your script does not know what to wait for.

You could try to communicate the PIDs up to the Perl script, but that quickly gets out of hand. Use fork().

Comments

[Chas. Owens]

You aren't checking for a failed fork and you are pushing the bareword pid onto @pids, not $pid. The exit should probably be a die instead, if it executes that means the exec failed.

[Dave]

Fixed. Thanks, I'm not really a native speaker of Perl.

[zdim]

This is a much better answer.

[Kjetil S.]

$pid == -1 should probably be !defined($pid) instead.

[Pratyush Rathore]

Apparently doing this creates a lot of zombie processes. en.wikipedia.org/wiki/Zombie_process

[Chas. Owens]

@PratyushRathore It shouldn't. A zombie results when the parent process doesn't call wait or waitpid. As you can see the parent is calling waitpid for a child that has exited. If you are getting a bunch of zombies then either you are doing something wrong or one of the first children is taking a long time and the later children have exited (but not been reaped). You could switch to while (waitpid(-1, WNOHANG) > 0) { sleep 1 } to reap the next exited child.

[Pratyush Rathore]

"one of the first children is taking a long time and the later children have exited (but not been reaped)." Yes, that is exactly what was happening in my case. Obviously, this was a very very special case with a lot of processes. I think, I wrote the last comment in a pretty bad style. I just wanted to ensure that people have a context for zombie processes when they are doing this.