What exactly is <() in bash (and =() in zsh)?

Question

I'm pretty comfortable with bash, but recently I ended up in a substitution I didn't know.

What exactly is <(command) in bash? How does it compare to the =(command) in zsh?

I understand that this has something to do with default file descriptors. In my computer

echo <()

returns /proc/self/fd/11, which I found out to be a copy of the script STDOUT, but this still seems pretty confusing to me.

Solution 1

This is called process substitution.

The <(list) syntax is supported by both, bash and zsh. It provides a way to pass the output of a command (list) to another command when using a pipe (|) is not possible. For example when a command just does not support input from STDIN or you need the output of multiple commands:

diff <(ls dirA) <(ls dirB)

---

<(list) connects the output of list with a file in /dev/fd, if supported by the system, otherwise a named pipe (FIFO) is used (which also depends on support by the system; neither manual says what happens if both mechanisms are not supported, presumably it aborts with an error). The name of the file is then passed as argument on the command line.

---

zsh additionally supports =(list) as possible replacement for <(list). With =(list) a temporary file is used instead of file in /dev/fd or a FIFO. It can be used as a replacement for <(list) if the program needs to lseek in the output.

According to the ZSH manual there might also be other issues with how <(list) works:

The = form is useful as both the /dev/fd and the named pipe implementation of <(...) have drawbacks. In the former case, some programmes may automatically close the file descriptor in question before examining the file on the command line, particularly if this is necessary for security reasons such as when the programme is running setuid. In the second case, if the programme does not actually open the file, the subshell attempting to read from or write to the pipe will (in a typical implementation, different operating systems may have different behaviour) block for ever and have to be killed explicitly. In both cases, the shell actually supplies the information using a pipe, so that programmes that expect to lseek (see man page lseek(2)) on the file will not work.

Solution 2

Note, this is a bash answer, not zsh.

There are cases in bash where you can't use pipes:

some_command | some_other_command

because pipes introduce subshells for each component of the pipeline, when the subshells exit, any side-effects you were relying on would disappear. For example, this contrived example:

cat file | while read line; do ((count++)); done
echo $count

will display a blank line, because the $count variable does not exist in the current shell.

A bash process substitution allows you to avoid this conundrum by allowing you to read from the "some_command" output like you would from a file

while read line; do ((count++)); done < <(cat file)
# ....................................1.2
echo $count   # the variable *does* exist in the current shell

(1) is a normal input redirection. (2) is the start of the <() process substitution syntax.

Solution 3

Another difference between =(command) & <(command) in zsh is synchronous vs asynchronous execution:

date & \
diff =(sleep 4; date) =(sleep 5; date) & \
diff <(sleep 4; date) <(sleep 5; date) &

Sun 29 Nov 2020 08:16:01 AEDT
[1]    41717 done       date

$ 1c1
< Sun 29 Nov 2020 08:16:05 AEDT
---
> Sun 29 Nov 2020 08:16:06 AEDT

[3]  + 41719 exit 1     diff <(sleep 4; date) <(sleep 5; date)

$ 1c1
< Sun 29 Nov 2020 08:16:05 AEDT
---
> Sun 29 Nov 2020 08:16:10 AEDT

[2]  + 41718 exit 1     diff =(sleep 4; date) =(sleep 5; date)

... although no idea if that was by design or not, as I can't think of a compelling reason to have =() function synchronously.

Comments

[Gombai Sándor]

=(cmdlist) in zsh has almost the same effect as <(cmdlist) in bash but it creates (and deletes when ready) a temporary file with the output of cmdlist for the redirection. This is good when seeking is potentially done in the program. <(cmdlist) is known by zsh as well.

[johnnyB]

This helped me figure out why MacOS pfctl -f <(echo "pf rules") would say bad file descriptor. using zsh and =(echo "pf rules") instead works.

[Elijah Lynn]

Anyone know what this is called or how to do this same thing in fish? I get "invalid redirection target" in fish shell with <(command) format.

[Adaephon]

@ElijahLynn It is called "process substitution". (Also, see the first line of the answer ;-) ). In fish this can be achieved with psub.

[Elijah Lynn]

I thought process substitution is the (command) part. Or does that include the < prefix as well or is there another name for that part?

[user1686]

@ElijahLynn: It includes the prefix; <( acts as a different token from lone <. Process substitution doesn't even behave like redirection (it doesn't alter the 'primary' command's stdin or stdout, but expands into a word) -- so it is definitely not the same as just having a < followed by ( ).

[dave_thompson_085]

In bash 4.2 up (released 2011, though not always used immediately) if shopt lastpipe is on and job control is inactive (usually true for script but not interactive, but can be changed) and the pipeline is not backgrounded, the last/rightmost command (only) runs in main shell not subshell and var settings etc are preserved. Other shells may have this as an option, or even automatic.

[gnom1gnom]

Can someone please explain why is the additional < required, and <( is not enough if it already redirects the stout of a command to a /dev/fd file? For example why head -c1 <(echo test) works fine and read x <(echo test) not?

[gnom1gnom]

I figured out the answer to my question: head takes a file parameter, while read takes stdin behind the scene, after process substitution, these commands look as follows head -c1 /dev/fd/63 read x < /dev/fd/63

[theonlygusti]

Is there a way to give a custom filename/extension to the <(...) file

[TBBle]

The difference is possibly that =() needs to write the program output to a temporary file, so it waits for program completion to take ownership of the output file, while <() just hooks up stdout of the sub-program as a pipe/FIFO input, and so must be asynchronous or it will block forever waiting for a reader, which is waiting for it, as described in the zsh manual quote in @Adaephon's answer above.

[Steve Wills]

The contrived example can be done in pure Bourne shell using here docs instead, which is much more portable and simpler and easier to understand.