How to make a function available to the command `parallel` (GNU)?

Question

In Bash, let's consider a function that does nothing but echo the argument followed by "is an integer".

f () { num="${!1}"; echo $num is an integer; }
number=12
f number
# 12 is an integer

I would like to write on a file a number of commands that uses the function f and then run these commands in parallel using the function parallel (GNU).

# Write Commands to the file `Commands.txt`
rm Commands.txt
touch Commands.txt
for i in $(seq 1 5)
do
   echo "number=$i; f number" >> Commands.txt
done

With source everything work fine

source Commands.txt
1 is an integer
2 is an integer
3 is an integer
4 is an integer
5 is an integer

However, when I try to run the commands in parallel it returns that the function f is not found

parallel :::: Commands.txt
/bin/bash: f: command not found
/bin/bash: f: command not found
/bin/bash: f: command not found
/bin/bash: f: command not found
/bin/bash: f: command not found

Is there a way I can make my function f available for parallel without having to define the function at each line of the file Commands.txt?

Solution 1

You basically have three options:

1. export -f (which is a non-POSIX bash feature)
2. Execute a shell defining the function on each invocation
3. Move the function to a shell script and run that instead

Option 1 is probably the easiest to demonstrate, so I'll show that one:

$ f() { num=$1; echo "$num is an integer"; }
$ export -f f
$ cat Commands.txt 
number=1; f "$number" 
number=2; f "$number" 
number=3; f "$number" 
number=4; f "$number" 
number=5; f "$number" 
$ parallel :::: Commands.txt
1 is an integer
2 is an integer
3 is an integer
4 is an integer
5 is an integer

Note that your population of Commands.txt is likely in error, you need f "$number" in Commands.txt to pass the number, not f number which would pass the literal string "number". Your script to generate it should do echo "number=$i; f \"\$number\"" (note the escapes, which are important to avoid $number being interpreted at echo time, or the "s terminating the string).

Solution 2

Bash can export functions through environment variables.

export -f f

Note that this is a bash feature, not available in other shells of the sh family.

Alternatively, make the function a script. Scripts and functions have the same syntax apart from the function definition line.

#!/bin/bash
num="${!1}"; echo $num is an integer;

If you do that, you'll need to export the variable you pass to the script: the lines in Command.txt will need to look like

export number=1; f number

or

number=1 f number

Comments

[Ole Tange]

Option 4: use env_parallel (described in man parallel) and use that to export the whole environment.

[clerksx]

@dave_thompson_085 Sure, but judging from the OPs expected output, it seems they want ${1}, not ${!1}.