Pass input file through pipe as argument?

Question

I have two issues. I am trying to pass files through a pipe as an argument, and I am trying to use that file as a variable inside the sh function.

Here is my command:

find . -name 'segment*' | xargs -n1 -P4 sh someFunction.sh

Here, my line is finding all files that look like "segment.something" and passing it into the right side of the pipe. In someFunction.sh, I need the name of the file as an argument. Let's say the file being inputted in segment1.

The someFunction.sh will print out every line of segment1 (for instance).

How do I pass output from the left hand side of the pipe to the right hand side, and then how do I call it within someFunction.sh?

Solution 1

Your command

$ find . -name 'segment*' | xargs -n1 -P4 sh someFunction.sh

has the effect that at most four copies of the someFunction.sh shell script will be started (-P 4) in parallel (new ones will be spawed as soon as the old ones are done), each one getting one filename as its argument (-n 1).

This means that each invocation of your script will look like

sh someFunction.sh segmentsomething

Inside the script, the shell will put the values of the positional parameters (the arguments on the command line) into $1, $2 etc. ($0 usually contains the name of the script itself). In your case, $1 will contain the name of the file, and the others will be empty.

So, in the script:

filename="$1"

echo "$filename"
cat "$filename"

That's that. Now, usually when one uses find to look for files and pass their filenames to xargs there is the issue with wonky filenames that people tend to remind each other of, and I'll do that here too.

The find utility passes filenames separated by whitespace. That's no good if you have filenames with spaces in them as that would cause problems for xargs to invoke your script with proper names.

Therefore, it's good practice to always use -print0 with find and -0 with xargs, which means that the filenames, instead of being space-separated, are separated by nul characters (\0). This makes it a lot safer.

Thus:

$ find . -name 'segment*' -print0 | xargs -0 -n1 -P4 sh someFunction.sh

Solution 2

You could use something like this assuming someFunction.sh is in your working directory.

find . -name 'segment*' -print0| xargs -0 -n1 -P4 ./someFunction.sh 

The -print0 and -0 allow for files with spaces in the name (A common problem). In my someFunction.sh I have

#!/bin/bash
echo "Arg: " $1  
cat $1

Which simply echo's out the file name then writes the contents of the file passed to someFunction.sh

Comments

[Kusalananda]

-print0 is missing in example.