Why does shell Command Substitution gobble up a trailing newline char?

shell text-processing command-substitution

6,099

Solution 1

Because the shell was not originally intended to be a full programming language.

It is quite difficult to remove a trailing \n from some command output. However, for display purposes, almost all commands end their output with \n, so… there has to be a simple way to remove it when you want to use it in another command. Automatic removal with the $() construction was the chosen solution.

So, maybe you'll accept this question as an answer:

Can you find a simple way to remove the trailing \n if this was not done automatically in the following command?

> echo The current date is "$(date)", have a good day!

Note that quoting is required to prevent smashing of double spaces that may appear in formatted dates.

Solution 2

It's part of the standard:

The shell shall expand the command substitution by executing command in a subshell environment (see Shell Execution Environment) and replacing the command substitution (the text of command plus the enclosing "$()" or backquotes) with the standard output of the command, removing sequences of one or more <newline>s at the end of the substitution.

Of course, the standard is probably written this way because that's how ksh did it or something, but it is the standard, and it is documented behavior. If you don't like it, use Perl or something else that will let you keep the trailing newlines.

Solution 3

Well, it makes sense to me. The newline is only there in the first place, in normal command output, so that the prompt appears after the command completes on a new line. The newline isn't part of the original output in most cases, it's there to tidy the screen up. When you're parsing output from a command the newline at the end is usually troublesome. Why does the wc command output 2 lines of text? Oh, it doesn't, it outputs one followed by a newline. When you parse wc you don't want to be worrying about the fact that there's 2 lines of output - there aren't really, there's only one.

Solution 4

I was running into same issue, and found below example. Seems quoting helped the situation, at least it did for me:

http://tldp.org/LDP/abs/html/commandsub.html.

dir_listing=`ls -l`
echo $dir_listing     # unquoted

# Expecting a nicely ordered directory listing.

# However, what you get is:
# total 3 -rw-rw-r-- 1 bozo bozo 30 May 13 17:15 1.txt -rw-rw-r-- 1 bozo
# bozo 51 May 15 20:57 t2.sh -rwxr-xr-x 1 bozo bozo 217 Mar 5 21:13 wi.sh

# The newlines disappeared.


echo "$dir_listing"   # quoted
# -rw-rw-r--    1 bozo       30 May 13 17:15 1.txt
# -rw-rw-r--    1 bozo       51 May 15 20:57 t2.sh
# -rwxr-xr-x    1 bozo      217 Mar  5 21:13 wi.sh

View more solutions

6,099

Peter.O

Updated on September 18, 2022

Comments

Peter.O almost 2 years
As per the following example, and as in my recent question In bash, where has the trailing newline char gone?, I want to know "why" it happens
```
  x="$(echo -ne "a\nb\n")" ; echo -n "$x" | xxd -p 
# Output is: 610a62 
# The trailing newline from the 'echo' command
#   has been "deleted" by Command Substitution
```
I assume there must be some very significant reason for a shell action, namely Command Substitution, to actually delete some data from the command output it is substituting...
but I can't get my head around this one, as it seems to be the antithesis of what it is supposed to do.. ie. to pass the output of a command back into the script process... Holding back one character seems weird to me, but I suppose there is a sensible reason for it... I'm keen to find out what that reason is...
Peter.O almost 13 years

@Philomath: Both x="hello " and x="hello " will lose all their trailing spaces if the are displayed via echo $x... However only the very last newline of the Command Substituton is lost.
Philomath almost 13 years

strange, I don't see any difference in my bash (verifying with xxd)
Peter.O almost 13 years

I just checke on this... It does remove all trailing newlines... I was experimenting with IFS when I got that impression, so let's cancel that one :).. but the question still remains... It is an fundamental part word splitting to condense multiple spaces to a single space, and to totally remove leading and trailing whitespace.. I can understand that, but I certainly don't understand why, with Command Substitution, it only strips \n and not all whitespace (as with word splitting), and why only the trailing end?.. and above all: "Command Substitution" is only partially substituting.. Why?
Peter.O almost 13 years

@EightBitTony: My question is not in any way related to displaying something in the terminal.. That is quite straight forward. If there is a newline to display, then it will display the newline.. The point in question here, is that a \n in the original stdout stream is removed by Command Substitution. ie. my original data ( very important data ) has trailing newlines removed, even when the data is wrapped in "quotes". I reasonably understand how and why Word Splitting works, but I have absolutely no idea why Command Substitution strips only newlines and only trailing ones.
EightBitTony almost 13 years

Nothing you have said changes my view. More often than not, the final newline in any output is there purely for display purposes, not as part of the data.
Peter.O almost 13 years

If what you say is true, then I would be absolutely stunned. If there is a shopt to change your described default behaviour, then I'd be happy again... I find it difficult to think that Bash/Linux/Unix would pollute such a critical function as "capturing stdout" just because date doesn't have a with/without '\n' option... The obvious fix would be that bash (or other shell) has a shopt for this... Do you know of any? .. and do you have any reference links that speak of the whys and wherefores of this isssue? ... and why doesn't date have a \n option.. It should!
Peter.O almost 13 years

I agree with your view, and have all along... Yes the newline at the end of output is for convenience... but my issue has nothing to do with that... I am asking: Why does Command Substitution forcibly remove it? ... These are two different issues.. Maybe my question should be: Is there a built in workaround? . because having to code for this issue by adding a trailing dummy char, sucks! ... I'll do it if I have to, but this is the first time I've been stymied by bash (and I'm in a state of shock :).. It does so much so well...
Stéphane Gimenez almost 13 years

Command substitutions appeared in the first version of the Bourne shell in the "7th edition" of Unix in 1979. It was already a big revolution and I guess (I was not born) that nobody cared about keeping trailing '\n' or not. Yes, you can read the manpage in less than 10min, and it seems nothing has changed about command substitution until then. Except for the preferred $() alternative syntax.
EightBitTony almost 13 years

As mentioned in another answer, it's part of the standard. For me, I think it does it that way because when you post-process data you only want to see the data, not the convenient newline. It's because the shell expects you to be handling the data in a pipe, and the newline is not data. Any more and we need to take this to a discussion.
Peter.O almost 13 years

Thanks.. I thought this may be a legacy issue, and it certainly is shaping up to the fact that I'd better start watching my Command Substitutions more carefully. Until today I must have been surviving on a wing an a prayer.. Some of those "mysterious happenings" I've encountered are starting to make sense now :) ... So in a reverse case of your "date" posit, if I want to keep my trailing \n, I must code something like this :( { echo -n "The current date is "; var="$(date; echo -e x)"; var="${var%x}"; echo -n "$var"; echo ", have a good day!"; } .. so be it :)
Peter.O almost 13 years

Thanks.. I've pretty well got the idea now.. It seems to be a case of Command Substitution simply leans towards dropping the trailing newlines, rather than maintaing them.. There must be some wisdom in that move which I havent "felt" yet, but I used to think that the overwhelming popular use of all lo-case letters in filenames was a bit strange/unnecessary, but I was thinking just the other day that it is actually quite a comfortable system.. I'll probably see (more deeply) the rhyme and reason of Command Substitution's behaviour, some day...
Peter.O almost 13 years

A nice summary.. Thanks.. and the links certainly helped
Peter.O almost 13 years

PS re my above comment: Of course, just date would work, but I just used date as an example of any command..
Gilles 'SO- stop being evil' almost 13 years

IFS has nothing to do with stripping newlines at the end of command substitutions.
Gilles 'SO- stop being evil' almost 13 years

The standard is like that because that's how the Bourne shell did it and every other shell since.
Peter.O almost 13 years

Your 'question' about date was the strongest promt towards my understanding of 'why' Command Substution does what it does, so this has been the 'best' answer to my question... and I certainly also needed the other good answers as well..
Admin about 11 years

The problem that the original poster is having is that the trailing newlines (just the very last ones) of his commands are disappearing. Unless passed the -n flag, echo adds a trailing newline to its output, so both of the echos in your example have trailing newlines. However, if you try echo -n $dir_listing or echo -n "$dir_listing", you will see that no trailing newline is printed with or without quotes around $dir_listing, which suggests that the problem is with the command substitution.
sancho.s ReinstateMonicaCellio almost 6 years

stackoverflow.com/questions/613572/…
user414777 over 3 years

"Because the shell was not originally intended to be a full programming language." [citation needed]