paste without temporary files in Unix

Question

I'm trying to use the Unix command paste, which is like a column-appending form of cat, and came across a puzzle I've never known how to solve in Unix.

How can you use the outputs of two different programs as the input for another program (without using temporary files)?

Ideally, I'd do this (without using temporary files):

./progA > tmpA; ./progB > tmpB; paste tmpA tmpB

This seems to come up relatively frequently for me, but I can't figure out how to use the output from two different programs (progA and progB) as input to another without using temporary files (tmpA and tmpB).

For commands like paste, simply using paste $(./progA) $(./progB) (in bash notation) won't do the trick, because it can read from files or stdin.

The reason I'm wary of the temporary files is that I don't want to have jobs running in parallel to cause problems by using the same file; ensuring a unique file name is sometimes difficult.

I'm currently using bash, but would be curious to see solutions for any Unix shell.

And most importantly, am I even approaching the problem in the correct way?

Cheers!

Solution 1

You do not need temp files under bash, try this:

paste <(./progA) <(./progB)

See "Process Substitution" in the Bash manual.

Solution 2

Use named pipes (FIFOs) like this:

mkfifo fA
mkfifo fB
progA > fA &
progB > fB &
paste fA fB
rm fA fB

The process substitution for Bash does a similar thing transparently, so use this only if you have a different shell.

Solution 3

Holy moly, I recent found out that in some instances, you can get your process substitution to work if you set the following inside of a bash script (should you need to):

set +o posix

http://www.linuxjournal.com/content/shell-process-redirection

From link: "Process substitution is not a POSIX compliant feature and so it may have to be enabled via: set +o posix" I was stuck for many hours, until I had done this. Here's hoping that this additional tidbit will help.

Comments

[ephemient]

Actually, Bash uses anonymous pipes and substitutes /dev/fd/## on platforms that support it -- it doesn't create temporary named pipes on Linux.

[Colin D Bennett]

This doesn't do a true paste, which would combine the lines in a side-by-side fashion. In the case of this command the output of progA and progB are concatenated. You might as well just omit the | paste since it is as useless as | cat