How do I bring HEREDOC text into a shell script variable?

shell-script posix

9,136

Solution 1

The problem is that, in Bash, inside $( ... ) escape (and other) sequences get parsed, even though the heredoc itself wouldn't have them. You get a doubled line because \ escapes the line break. What you're seeing is really a parsing issue in Bash - other shells don't do this. Backticks can also be a problem in older versions. I have confirmed that this is a bug in Bash, and it will be fixed in future versions.

You can at least simplify your function drastically:

func() {
    res=$(cat)
}
func <<'HEREDOC'
...
HEREDOC

If you want to choose the output variable it can be parameterised:

func() {
    eval "$1"'=$(cat)'
}
func res<<'HEREDOC'
...
HEREDOC

Or a fairly ugly one without eval:

{ res=$(cat) ; } <<'HEREDOC'
...
HEREDOC

The {} are needed, rather than (), so that the variable remains available afterwards.

Depending on how often you'll do this, and to what end, you might prefer one or another of these options. The last one is the most concise for a one-off.

If you're able to use zsh, your original command substitution + heredoc will work as-is, but you can also collapse all of this down further:

x=$(<<'EOT'
...
EOT
)

Bash doesn't support this and I don't think any other shell that would experience the problem you're having does either.

Solution 2

About the OP solution:

You do not need an eval to assign a variable if you allow some constant variable to be used.
the general structure of calling a function that receives the HEREDOC could also be implemented.

A solution that works in all (reasonable) shells with both items solved is this:

#!/bin/bash
nl="
"

read_heredoc(){
    var=""
    while IFS="$nl" read -r line; do
        var="$var$line$nl"
    done 
}


read_heredoc <<'HEREDOC'

                        _                            _ _
                       | |                          | (_)
  _ __ ___  _   _ _ __ | | __ _  ___ ___  ___  _ __ | |_ _ __   ___
 | '_ ` _ \| | | | '_ \| |/ _` |/ __/ _ \/ _ \| '_ \| | | '_ \ / _ \
 | | | | | | |_| | |_) | | (_| | (_|  __/ (_) | | | | | | | | |  __/
 |_| |_| |_|\__, | .__/|_|\__,_|\___\___|\___/|_| |_|_|_|_| |_|\___|
             __/ | |
            |___/|_|

HEREDOC

read_heredoc2_result="$str"

printf '%s' "${read_heredoc2_result}"

A solution for the original question.

A solution that works since bash 2.04 (and recent zsh, lksh, mksh).
Look below for a more portable (POSIX) version.

#!/bin/bash
read_heredoc() {
    IFS='' read -d '' -r var <<'HEREDOC'

                        _                            _ _
                       | |                          | (_)
  _ __ ___  _   _ _ __ | | __ _  ___ ___  ___  _ __ | |_ _ __   ___
 | '_ ` _ \| | | | '_ \| |/ _` |/ __/ _ \/ _ \| '_ \| | | '_ \ / _ \
 | | | | | | |_| | |_) | | (_| | (_|  __/ (_) | | | | | | | | |  __/
 |_| |_| |_|\__, | .__/|_|\__,_|\___\___|\___/|_| |_|_|_|_| |_|\___|
             __/ | |
            |___/|_|



HEREDOC

}

read_heredoc
echo "$var"

The core command

IFS='' read -d '' -r var <<'HEREDOC'

works as follows:

The word HEREDOC is (single) quoted to avoid any expansion of the text that follows.
The "here doc" contents are served in the stdin with <<.
The option -d '' forces read to slurp the whole content of the "here doc".
The -r option avoids interpretation of backslash quoted characters.
The core command is similar to read var.
And the last detail is IFS='', which will avoid that read remove leading or trailing characters in the default IFS: spacetabnewline.

In ksh, the null value for the -d '' option doesn't work.
As a workaround, if the text has no "carriage return", a -d $'\r' works (if a $'\r' is added to the end of each line, of course).

An added (in comments) requirement is to generate a POSIX compliant solution.

POSIX

Extending the idea to make it run only with POSIX options.
That means mainly no -d for read. That forces a read for each line.
That, in turn forces the need to capture a line at a time.
Then, to build the var a trailing new line must be added (as the read removed it).

#!/bin/sh

nl='
'

read_heredoc() {
    unset var
    while IFS="$nl" read -r line; do
        var="$var$line$nl"
    done <<\HEREDOC

                        _                            _ _
                       | |                          | (_)
  _ __ ___  _   _ _ __ | | __ _  ___ ___  ___  _ __ | |_ _ __   ___
 | '_ ` _ \| | | | '_ \| |/ _` |/ __/ _ \/ _ \| '_ \| | | '_ \ / _ \ 
 | | | | | | |_| | |_) | | (_| | (_|  __/ (_) | | | | | | | | |  __/ 
 |_| |_| |_|\__, | .__/|_|\__,_|\___\___|\___/|_| |_|_|_|_| |_|\___| 
             __/ | | 
            |___/|_| 



HEREDOC

}

read_heredoc
printf '%s' "$var"

That works (and has been tested) in all reasonable shells.

Solution 3

Useless use of cat (quote \ and `):

myplaceonline="
                       _                            _ _            
 _ __ ___  _   _ _ __ | | __ _  ___ ___  ___  _ __ | (_)_ __   ___ 
| '_ \` _ \\| | | | '_ \\| |/ _\` |/ __/ _ \\/ _ \\| '_ \\| | | '_ \\ / _ \\
| | | | | | |_| | |_) | | (_| | (_|  __/ (_) | | | | | | | | |  __/
|_| |_| |_|\\__, | .__/|_|\\__,_|\\___\\___|\\___/|_| |_|_|_|_| |_|\\___|
       |___/|_
"

Or without quoting:

myplaceonline="$(figlet myplaceonline)"

Solution 4

To support trailing newlines, I combined the answer from @MichaelHomer and my original solution. I didn't use the suggested workarounds from the link that @EliahKagan noted because the first one uses magical strings and the last two weren't POSIX compliant.

#!/bin/sh

NEWLINE="
"

read_heredoc() {
  read_heredoc_result=""
  while IFS="${NEWLINE}" read -r read_heredoc_line; do
    read_heredoc_result="${read_heredoc_result}${read_heredoc_line}${NEWLINE}"
  done
  eval $1'=${read_heredoc_result}'
}

read_heredoc heredoc_str <<'HEREDOC'

                        _                            _ _
                       | |                          | (_)
  _ __ ___  _   _ _ __ | | __ _  ___ ___  ___  _ __ | |_ _ __   ___
 | '_ ` _ \| | | | '_ \| |/ _` |/ __/ _ \/ _ \| '_ \| | | '_ \ / _ \
 | | | | | | |_| | |_) | | (_| | (_|  __/ (_) | | | | | | | | |  __/
 |_| |_| |_|\__, | .__/|_|\__,_|\___\___|\___/|_| |_|_|_|_| |_|\___|
             __/ | |
            |___/|_|




HEREDOC

echo "${heredoc_str}"

View more solutions

9,136

Kevin

Updated on September 18, 2022

Comments

Kevin almost 2 years

I'm trying to bring HEREDOC text into a shell script variable in a POSIX compliant way. I tried like so:

#!/bin/sh

NEWLINE="
"

read_heredoc2() {
  while IFS="$NEWLINE" read -r read_heredoc_line; do
    echo "${read_heredoc_line}"
  done
}

read_heredoc2_result="$(read_heredoc2 <<'HEREDOC'

                        _                            _ _
                       | |                          | (_)
  _ __ ___  _   _ _ __ | | __ _  ___ ___  ___  _ __ | |_ _ __   ___
 | '_ ` _ \| | | | '_ \| |/ _` |/ __/ _ \/ _ \| '_ \| | | '_ \ / _ \
 | | | | | | |_| | |_) | | (_| | (_|  __/ (_) | | | | | | | | |  __/
 |_| |_| |_|\__, | .__/|_|\__,_|\___\___|\___/|_| |_|_|_|_| |_|\___|
             __/ | |
            |___/|_|



HEREDOC
)"

echo "${read_heredoc2_result}"

That produced the following which is wrong:

                        _                            _ _
                       | |                          | (_)
  _ __ ___  _   _ _ __ | | __ _  ___ ___  ___  _ __ | |_ _ __   ___
 | '_ ` _ \| | | | '_ \| |/ _` |/ __/ _ \/ _ \| '_ \| | | '_ \ / _  | | | | | | |_| | |_) | | (_| | (_|  __/ (_) | | | | | | | | |  __/
 |_| |_| |_|\__, | .__/|_|\__,_|\___\___|\___/|_| |_|_|_|_| |_|\___|
             __/ | |
            |___/|_|

The following works but I don't like how clunky it is by using a random output variable:

#!/bin/sh

NEWLINE="
"

read_heredoc1() {
  read_heredoc_first=1
  read_heredoc_result=""
  while IFS="$NEWLINE" read -r read_heredoc_line; do
    if [ ${read_heredoc_first} -eq 1 ]; then
      read_heredoc_result="${read_heredoc_line}"
      read_heredoc_first=0
    else
      read_heredoc_result="${read_heredoc_result}${NEWLINE}${read_heredoc_line}"
    fi
  done
}

read_heredoc1 <<'HEREDOC'

                        _                            _ _            
                       | |                          | (_)           
  _ __ ___  _   _ _ __ | | __ _  ___ ___  ___  _ __ | |_ _ __   ___ 
 | '_ ` _ \| | | | '_ \| |/ _` |/ __/ _ \/ _ \| '_ \| | | '_ \ / _ \
 | | | | | | |_| | |_) | | (_| | (_|  __/ (_) | | | | | | | | |  __/
 |_| |_| |_|\__, | .__/|_|\__,_|\___\___|\___/|_| |_|_|_|_| |_|\___|
             __/ | |                                                
            |___/|_|                                                



HEREDOC

echo "${read_heredoc_result}"

Correct output:

                        _                            _ _            
                       | |                          | (_)           
  _ __ ___  _   _ _ __ | | __ _  ___ ___  ___  _ __ | |_ _ __   ___ 
 | '_ ` _ \| | | | '_ \| |/ _` |/ __/ _ \/ _ \| '_ \| | | '_ \ / _ \
 | | | | | | |_| | |_) | | (_| | (_|  __/ (_) | | | | | | | | |  __/
 |_| |_| |_|\__, | .__/|_|\__,_|\___\___|\___/|_| |_|_|_|_| |_|\___|
             __/ | |                                                
            |___/|_|

Any ideas?

Kusalananda over 7 years

If the banner is used only once, use cat with a here-document directly. If it's used in many places in the script, store it on file an cat it from there, just like /etc/motd is used on some systems.
Michael Homer over 7 years

Note that the problem you're having is actually a Bash bug - you had a POSIX solution already in your original attempt, which will work fine in ksh, dash, ash, and the oldest Bourne shell I can find. Bash command substitution parsing is weird, and used to be even more broken.
Kevin over 7 years

@MichaelHomer Oh, interesting! Wow, I'm on the latest Fedora 25, bash 4.3.43-4.fc25
Michael Homer over 7 years

Yeah. If you can get a 3-series version to try the script with, it breaks even sooner at the first backtick, so it's improving. I'm not sure whether they consider it a bug - arguably POSIX doesn't explicitly prohibit this behaviour, but it's fairly clear that command substitution contains all the characters until the ) and that quoted heredocs don't have expansions.
Michael Homer over 7 years

I've asked a question to try to sort out what POSIX really requires in this situation.
Kevin over 7 years

@MichaelHomer Cool, thanks. Great research!
done over 7 years

@Kevin I have updated my answer (at the start) perhaps it will be useful.

Kevin over 7 years

@EliahKagan Thanks, that might help, I'll try those out.
Kevin over 7 years

@EliahKagan I've added a new answer below.
Kevin over 7 years

@sorontar I just tested it and the trailing new line didn't seem erased to me. I'm not sure what you mean by the first point about using a variable to read the first line. Regarding the eval, it's only used for the name of the "output" variable. If we assume trusted users of the function, are there any other issues using eval in this example?
done over 7 years

Yes, A here-document must end with a newline, and you code remove it. Every "text line must end on a newline". Search in definitions of text file.
done over 7 years

3.206 Line A sequence of zero or more non- <newline> characters plus a terminating <newline> character.
Kevin over 7 years

@sorontar That makes sense, thanks for the references!
phk over 7 years

Latter is not POSIX.
ctx over 7 years

you are right, now it should be?
Kevin over 7 years

@ctx Thanks for the answer, but an unstated requirement of mine is that I want to be able to bring in unquoted HEREDOCs (partly because I'm making this into a public API function for a library called posixcube). The accepted answer is ultimately correct that there is a Bash bug processing nested HEREDOCs in command substitution (see the comment from Michael Homer in the question which links to another question which links to the Bash mailing list where the bug is confirmed by a Bash maintainer). My answer above on Jan 29th is my workaround solution and works well.
saulius2 over 3 years

That backslash in the<<\HEREDOC ! Thanks a million.