How to avoid bash command substitution to remove the newline character?

Question

To speed up some bash script execution, I would like to keep the result of a command in a variable using command substitution, but the command substitution replaces the 0x0A newline character by a space. For example:

a=`df -H`

or

a=$( df -H )

When I want to process further $a, the newline characters are replaced by a space and all the lines are now on one line, which is much harder to grep:

echo $a

What would be the easy tricks to avoid the newline character being removed by the command substitution?

Solution 1

Non-trailing newlines are not removed

The newlines you are looking for are there, you just don't see them, because you use echo without quoting the variable.

Validation:

$ a=$( df -H )
$ echo $a
Filesystem Size Used Avail Use% Mounted on /dev/sda3 276G 50G 213G 19% / udev 2.1G 4.1k 2.1G 1% /dev tmpfs 832M 820k 832M 1% /run none 5.3M 0 5.3M 0% /run/lock none 2.1G 320k 2.1G 1% /run/shm
$ echo "$a"
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda3       276G   50G  213G  19% /
udev            2.1G  4.1k  2.1G   1% /dev
tmpfs           832M  820k  832M   1% /run
none            5.3M     0  5.3M   0% /run/lock
none            2.1G  320k  2.1G   1% /run/shm
$ 

Trailing newlines are removed

As @user4815162342 correctly pointed out, although newlines within the output are not removed, trailing newlines are removed with command substitution. See experiment below:

$ a=$'test\n\n'
$ echo "$a"
test


$ b=$(echo "$a")
$ echo "$b"
test
$

In most cases this does not matter, because echo will add the removed newline (unless it is invoked with the -n option), but there are some edge cases where there are more that one trailing newlines in the output of a program, and they are significant for some reason.

Workarounds

1. Add dummy character

In these case, as @Scrutinizer mentioned, you can use the following workaround:

$ a=$(printf 'test\n\n'; printf x); a=${a%x}
$ echo "$a"
test


$ 

Explanation: Character x is added to the output (using printf x), after the newlines. Since the newlines are not trailing any more, they are not removed by the command substitution. The next step is to remove the x we added, using the % operator in ${a%x}. Now we have the original output, with all newlines present!!!

2. Read using process substitution

Instead of using command substitution to assign the output of a program to variable, we can instead use process substitution to feed the output of the program to the read built-in command (credit to @ormaaj). Process substitution preserves all newlines. Reading the output to a variable is a bit tricky, but you can do it like this:

$ IFS= read -rd '' var < <( printf 'test\n\n' ) 
$ echo "$var"
test


$ 

Explanation:

• We set the internal field separator for the read command to null, with IFS=. Otherwise read would not assign the entire output to var, but only the first token.
• We invoke read with options -rd ''. The r is for preventing the backslash to act as a special character, and with d '' set the delimiter to nothing, so that read reads the entire output, instead of just the first line.

3. Read from a pipe

Instead of using command or process substitution to assign the output of a program to variable, we can instead pipe the output of the program to the read command (credit to @ormaaj). Piping also preserves all newlines. Note however, that this time we set the lastpipe shell optional behavior, using the shopt builtin. This is required, so that the read command is executed in the current shell environment. Otherwise, the variable will be assigned in a subshell, and it will not be accessible from the rest of the script.

$ cat test.sh 
#!/bin/bash
shopt -s lastpipe
printf "test\n\n" | IFS= read -rd '' var
echo "$var"
$ ./test.sh 
test


$

Solution 2

I was trying to wrap my head around this because I was using bash to stream in the result of running the interpreter on an F# script. After some trial and error, this turned out to solve the problem:

$ cat fsi.ch
#!/bin/bash
echo "$(fsharpi --quiet --exec --nologo $1)"

$ fsi.ch messages.fsx
Welcome to my program. Choose from the menu:
new | show | remove

Assuming, of course that you need to run a terminal program. Hope this helps.

Solution 3

Another "neat trick" is to use the carriage return character, which prevents the newline from being stripped but doesn't add anything to the output:

$ my_func_1 () {
>     echo "This newline is squashed"
> }
$ my_func_2 () {
>     echo "This newline is not squashed"
>     echo -n $'\r'
> }
$ echo -n "$(my_func_1)" && echo -n "$(my_func_2)" && echo done
This newline is squashedThis newline is not squashed
done
$

But buyer beware: as mentioned in the comments this can work nicely for output that is simply going to the terminal, but if you are passing this on to another process you might confuse it as it probably won't be expecting the weird terminating '\r'.

Comments

[enharmonic]

I have found that this issue also occurs when using eval.

[user4815162342]

The trailing newlines are still removed, though, and I'm afraid it cannot be avoided. (The OP probably doesn't care about those, but this is good to know in general.)

[Laurent]

Thank you very much, very enlightening - another example of focusing on the wrong place, since it wasn't the command substitution removing the newline.

[user000001]

@user4815162342 Thank you for the comment. I updated the answer for future readers.

[Scrutinizer]

@user4815162342: Interesting, a crummy work around might be to use a=$(printf 'test\n\n'; printf x); echo "${a%x}"

[ormaaj]

It isn't a very good workaround. There isn't a good way to avoid stripping trailing newlines with a command substitution. The best solution is to use read with nonstandard options to make the assignment. shopt -s lastpipe; printf %s "$myData" | IFS= read -rd '' var. Bash's printf -v is also useful in some similar situations.

[Scrutinizer]

@ormaaj: Why, in your opinion is it not a good workaround? I agree it is crummy, but what have you found that is not working properly?

[ormaaj]

@Scrutinizer It involves two assignments, and all that string manipulation just to work around the problem isn't very obvious. It doesn't have to be broken to be crap. A command substitution in most shells is just sugar for reading from a pipe to begin with, so working around the problem that way is much more direct. In practice, stripping trailing newlines is usually a desirable trait and I can't say I've had to use this in a real script more than a few times. (I can think of a case where this kludge may be preferable to read, but it doesn't apply to Bash.)

[user000001]

@ormaaj I added your workaround to the answer as well. Feel free to edit if my explanation is wrong in any way. I noticed though that although in a script it works fine, in the terminal it only outputs a blank like. I had to enclose it in ( ) to work in the terminal like this: ( shopt -s lastpipe; printf 'test\n\n' | IFS= read -rd '' var; echo "$var"; )

[ormaaj]

@user000001 It's much more common and backwards-compatible in Bash to use a process substitution in a redirect rather than rely on lastpipe (though I usually use lastpipe anyway). The reason you notice that is due to job control. Interactive shells have set -m which forces pipelines to run in separate process groups, which necessitates a sub-shell for the last element. Since job control doesn't apply to children of a subshell, wrapping everything in (...) is a workaround (which I almost always do when testing interactively anyways).

[AnyDev]

Note: read exits with non zero when it hits EOF. This causes methods 2 and 3 to exit with non-zero therefore masking main command's exit status. In Method 1 using main_cmd ; printf x also overwrites main commands exit code and interferes with error handling. Try main_cmd && printf x instead (if trailing lines are important only on success exit) or main_cmd ; ec=$?; printf x; exit $ec for keeping trailing spaces in both success and error cases.

[user000001]

@ormaaj Interestingly, one and a half year later, AndrDevEK found a disadvantage of the read method. It masks the exit status. Read the comment above.

[ormaaj]

@user000001 That is expected and not really a disadvantage. If you must access the status you can get it through the PIPESTATUS array. That will only work if you actually use a pipeline, as bash doesn't expose the exit status of process substitutions. shopt -s lastpipe; { printf %s $'foo\nbar'; exit 3; } | IFS= read -rd '' x; printf '%q, %s\n' "$x" "${PIPESTATUS[0]}"

[giorgiosironi]

"Always quote your variables, kids"

[user000001]

Nice trick if the intent is to only process the output visually and not programmatically. But if you look at the output of echo -n "$(my_func_2)" | cat -A you will see that the $'\r' is retained in the output, so it could cause problems if the receiver process doesn't expect it.