Multiline syntax for piping a heredoc; is this portable?

Question

I'm familiar with this syntax:

cmd1 << EOF | cmd2
text
EOF

but just discovered that bash allows me to write:

cmd1 << EOF |
text
EOF
cmd2

(the heredoc is used as input to cmd1, and the output of cmd1 is piped to cmd2). This seems like a very odd syntax. Is it portable?

Solution 1

Yes, the POSIX standard allows this. According to the 2008 version:

The here-document shall be treated as a single word that begins after the next <newline> and continues until there is a line containing only the delimiter and a <newline>, with no <blank> characters in between. Then the next here-document starts, if there is one.

And includes this example of multiple "here-documents" in the same line:

cat <<eof1; cat <<eof2
Hi,
eof1
Helene.
eof2

So there is no problem doing redirections or pipes. Your example is similar to something like this:

cat file |
cmd

And the shell grammar (further down on the linked page) includes these definitions:

pipe_sequence    :                             command
                 | pipe_sequence '|' linebreak command

newline_list     :              NEWLINE
                 | newline_list NEWLINE
                 ;
linebreak        : newline_list
                 | /* empty */

So a pipe symbol can be followed by an end-of-line and still be considered part of a pipeline.

Solution 2

Yes it's in the POSIX shell grammar. You can also have more than one here-doc for the same command (some other examples use two cat invocations, but this works as well):

cat <<EOF1 <<EOF2
first here-doc
EOF1
second here-doc
EOF2

This is contrived (using 2 here-docs for stdin), but if you think of providing input for different file descriptors it immediately makes sense.

There's also the possibility to drop the cat entirely. Why not make the here-document directly available to cmd:

cmd << EOF
input
here
EOF

Solution 3

Hmm, I suppose yes, according to the test in bash in POSIX mode:

$ bash --posix
$ cat <<EOF |
> ahoj
> nazdar
> EOF
> sed 's/a/b/'
bhoj
nbzdar

Solution 4

Hi, check this, for example

#!/bin/sh
( base32 -d | base64 -d )<<ENDOFTEXT
KNDWW42DNNSHS5ZXPJCG4MSVM5MVQVT2JFCTK3DELBFDCY2IIJYGE2JUJNHWS22LINVHQMCMNVFD
CWJQIIZVUV2JOVNEOVJLINTW6PIK
ENDOFTEXT

regards

View more solutions

Comments

[PaulC]

I came here to find a good way of splitting this into multiple lines: big-long-command1 with lots of args << EOF | big-long-command2 with lots of args. The "odd syntax" seems like the best way.

[Sridhar Sarnobat]

One convenient use case for this is when you're trying to convert a table that is space delimited into one that is tab-delimited so you can paste it in Google Spreadsheets. You won't have to create a temporary file.

[Sridhar Sarnobat]

The 1st one didn't work for me in z-shell. I don't like the 2nd one because it alienates the | from the command, losing the idiomacy (?) of shell pipelines.

[user1424739]

``` cat <<EOF1 <<EOF2 first here-doc EOF1 second here-doc EOF2 ``` The above does not work.

[Jens]

@user1424739 It does work in current zsh and bash. The ash and ksh93 seem to output only the second here doc.

[Jens]

Why the downvote? If there's something inaccurate, please give me opportunity to remedy.

[dragon788]

This is pretty sweet when using sudo tee /etc/securefile.conf <<EOF.

[Sridhar Sarnobat]

Just one other tiny note: do not put any spaces after the closing EOF. The prompt will behave strangely and you'll wonder what the hell is wrong

[Charles Duffy]

Running bash in POSIX-mode shuts off some extensions, but not by any means even nearly all of them. As such, while this answer is correct in terms of what POSIX allows, its reasoning doesn't support that very effectively.

[huelbois]

On what bash version does it work ? Using bash 4.4.19 (on ubuntu 18.04.02) and bash 5.0 (docker image), I only got the second here-doc. Or maybe there's a specific option ?

[Cigarette Smoking Man]

@huelbois Same here with bash 5.1.8