How to use fork() in Perl?

Question

I have hundreds of file names in an array. I want to create a child process for every 4 files in the array, and have that child do some stuff to each of those 4 files. (So with 100 files, I'll create 25 processes.)

I'm having some trouble understanding the order in which lines are processed when there's a fork. I was thinking I could do something like this, but I'm getting stuck:

foreach $file (@files) {
 if ($f++ % 4 == 0) {
  my $pid = fork();

  if ($pid) {
   push(@childs, $pid);
  }
  elsif ($pid == 0) {
    ... do stuff to $file ...
  }
 }

I don't think this is right, and I'm hoping someone can point me in the right direction. Thanks.

Solution 1

In addition to your trouble using fork, you also seem to have trouble partitioning your @files array into smaller sets of four files. Maybe something like this:

for (my $i = 0; $i < @files; $i += 4) {

    # take a slice of 4 elements from @files
    my @files4 = @files[$i .. $i + 3];

    # do something with them in a child process
    if (fork() == 0) {
        ... do something with @files4 ...
        exit;   # <--- this is very important
    }
}

# wait for the child processes to finish
wait for 0 .. @files/4;

Solution 2

Use Parallel::ForkManager

use Parallel::ForkManager qw( );

my $pm = Parallel::ForkManager->new(int(@files/4));
for my $file (@files) {
   my $pid = $pm->start and next;

   ... do something with $file ...

   $pm->finish; # Terminates the child process
}

Note that this still creates 100 processes, it simply limits it to 25 concurrent.

If you truly want only 25 processes, you can use the following:

use List::Util            qw( min );
use Parallel::ForkManager qw( );

my $pm = Parallel::ForkManager->new(0+@files);
while (@files) {
   my @batch = @files[0..min(4, $#files)];
   my $pid = $pm->start and next;

   for my $file (@batch) {
      ... do something with $file ...
   }

   $pm->finish; # Terminates the child process
}

Solution 3

i would group to an array, and let the child handle that group

my $group = []
foreach my $file (@files) {
    push @$group, $file;

    if(scalar(@$group) % 4 == 0) {
        my $pid = fork;
        die "Unable to fork!" unless defined $pid;
        push @childs, $pid if $pid;
        children_work($group) unless $pid;
        $group = [];
    }        
}

sub children_work {
   my $group = shift;

   // child, work with $group
   exit(0);
}

Comments

[ysth]

while (my @files4 = splice(@files, 0, 4)) { (destroys @files, though)

[itzy]

Since you're incrementing $i by 4, you don't need to multiply it by 4 when slicing, right?

[Eric Strom]

use List::Gen 'by'; for my $files4 (by 4 => @files) {do something with @$files4}