Is there any way to read lines from command output?

Question

I have a pre process command to output a file

./preprocess.sh > preprocessed_file 

and the preprocessed_file will be used like this

while read line
do

    ./research.sh $line &

done < preprocessed_file 

rm -f preprocessed_file

Is there any way to direct the output to the while read line part instead of outputting to the preprocessed_file? I think there should be a better way other than using this temp preprocessed_file.

Solution 1

You can use bash process substitution:

while IFS= read -r line; do
  ./research.sh "$line" &
done < <(./preprocess.sh)

Some advantages of process substitution:

• No need to save temporary files.
• Better performance. Reading from another process often faster than writing to disk, then read back in.
• Save time to computation since when it is performed simultaneously with parameter and variable expansion, command substitution, and arithmetic expansion

Solution 2

Yes! You can use a process pipe |.

./preprocess.sh |
    while IFS= read -r line
    do
        ./research.sh "$line" &
    done

A process pipe passes the standard output (stdout) of one process to the standard input (stdin) of the next.

You can optionally put a newline character following a | and extend the command to the next line.

Note: a|b is equivalent to b < <(a), but without the magic files, and in a more readable order, especially when the pipeline gets longer.

a|b|c is equivalent to c < <(b < <(a))

and

a|b|c|d|e is e < < (d < <(c < <(b < <(a))))

Comments

[Marcus Thornton]

what does the double left arrows (<<) mean?

[cuonglm]

@MarcusThornton: < is a redirection, while <(...) is process substitution syntax. You should read: gnu.org/software/bash/manual/html_node/… for more details.

[Marcus Thornton]

Got it. <(...) is a part of syntax.

[ctrl-alt-delor]

<(a) runs a sub-process with its output redirected to a magic file in /dev/fd. The preceding < takes it out of the file, as in your question. This magic file is automatically deleted: it saw never a real file anyway. Note that <(…) does not work on all systems, e.g. Cygwin a Gnu Operating system, that does not use Linux, but instead running within Microsoft-windows, using a dll as its kernel. The special/magic files have to be supported by the kernel.

[Stéphane Chazelas]

It's not necessarily faster. Because when reading from a pipe read has to read one byte at a time, while it can optimise things with reading larger chunks and seek backward when reading from a regular file. Best is to avoid while read loops altogether in the first place when possible. Also note that you need IFS= read -r line to read the line into $line. And leaving $line unquoted (invoking the split+glob operator) here probably doesn't make sense.

[vinc17]

Note: This solution with the pipe has the advantage to be more portable than process substitution (not supported by some POSIX shells like dash). Still concerning portability, the right hand side of a pipe may be executed in a subshell (this depends on the shell), so that any side effect (such as setting variables) may not affect the environment of the shell script.

[cuonglm]

I only focus on problem that OP had, do not have time to fix other problem, updated.

[mikeserv]

@StéphaneChazelas - why do you say a byte at a time? Depending on INPUT | while read whether INPUT's stdin is connected to a terminal, I would think it would be line-buffered. I'm only lately getting anything close to a grip on how that works, though.

[goldilocks]

@G-Man Possibly not in this context. If research.sh works with the command line argument array and $line is, e.g., "one two", with the intention that the first argument be "one" and the second argument "two", quoting $line will make that impossible -- instead the first argument will be "one two" and there won't be a second one...

[Digital Trauma]

"a|b is equivalent to b < <(a)" - close, but not quite. In the pipe version, both sides of the pipe are run in subshells, whereas in the process-substitution version, only the substituted process is run in a subshell, but a is run in the scope of the currently executing shell level. This has important implications for the scope of variables set within a

[Stéphane Chazelas]

@mikeserv, commands often line-buffer (as oppose to full-buffer) their output when it goes to a terminal. Here I'm saying that the read shell builtin reads one character at a time when reading from a pipe (regardless of what's at the other end of the pipe which read has no way to know), which is one of the reasons while read loops are tremendously slow.

[papo]

@Digital I guess you meant "This has important implications for the scope of variables set within b", though also 'a' runs in a subshell, there is more reason to store result in the part 'b' after being processed by 'a'.