Using “while read…”,echo and printf get different outcomes

Question

According to this question "Using "while read..." in a linux script"

echo '1 2 3 4 5 6' | while read a b c;do echo "$a, $b, $c"; done

outcome:

1, 2, 3 4 5 6

but when I replace echo with printf

echo '1 2 3 4 5 6' | while read a b c ;do printf "%d, %d, %d \n" $a $b $c; done

outcome

1, 2, 3
4, 5, 6

Could someone please tell me what makes these two commands different? Thanks~

Solution 1

It's not just echo vs printf

First, let's understand what happens with read a b c part. read will perform word-splitting based on the default value of IFS variable which is space-tab-newline, and fit everything based on that. If there's more input than the variables to hold it, it will fit splitted parts into first variables, and what can't be fitted - will go into last. Here's what I mean:

bash-4.3$ read a b c <<< "one two three four"
bash-4.3$ echo $a
one
bash-4.3$ echo $b
two
bash-4.3$ echo $c
three four

This is exactly how it is described in bash's manual (see the quote at the end of the answer).

In your case what happens is that, 1 and 2 fit into a and b variables, and c takes everything else, which is 3 4 5 6.

What you also will see a lot of times is that people use while IFS= read -r line; do ... ; done < input.txt to read text files line by line. Again, IFS= is here for a reason to control word-splitting, or more specifically - disable it, and read a single line of text into a variable. If it wasn't there, read would be trying to fit each individual word into line variable. But that's another story, which I encourage you to study later, since while IFS= read -r variable is a very frequently used structure.

echo vs printf behavior

echo does what you'd expect here. It displays your variables exactly as read has arranged them. This has been already demonstrated in previous discussion.

printf is very special, because it will keep on fitting variables into format string until all of them are exhausted. So when you do printf "%d, %d, %d \n" $a $b $c printf sees format string with 3 decimals, but there's more arguments than 3 (because your variables actually expand to individual 1,2,3,4,5,6). This may sound confusing, but exists for a reason as improved behavior from what the real printf() function does in C language.

What you also did here that affects the output is that your variables are not quoted, which allows the shell ( not printf ) to break down variables into 6 separate items. Compare this with quoting:

bash-4.3$ read a b c <<< "1 2 3 4"
bash-4.3$ printf "%d %d %d\n" "$a" "$b" "$c"
bash: printf: 3 4: invalid number
1 2 3

Exactly because $c variable is quoted, it is now recognized as one whole string, 3 4, and it doesn't fit the %d format, which is just a single integer

Now do the same without quoting:

bash-4.3$ printf "%d %d %d\n" $a $b $c
1 2 3
4 0 0

printf again says: "OK, you have 6 items there but format shows only 3, so I'll keep fitting stuff and leaving blank whatever I cannot match to actual input from user".

And in all these cases you don't have to take my word for it. Just run strace -e trace=execve and see for yourself what does the command actually "see":

bash-4.3$ strace -e trace=execve printf "%d %d %d\n" $a $b $c
execve("/usr/bin/printf", ["printf", "%d %d %d\\n", "1", "2", "3", "4"], [/* 80 vars */]) = 0
1 2 3
4 0 0
+++ exited with 0 +++

bash-4.3$ strace -e trace=execve printf "%d %d %d\n" "$a" "$b" "$c"
execve("/usr/bin/printf", ["printf", "%d %d %d\\n", "1", "2", "3 4"], [/* 80 vars */]) = 0
1 2 printf: ‘3 4’: value not completely converted
3
+++ exited with 1 +++

Additional notes

As Charles Duffy properly pointed out in the comments,bash has its own built-in printf, which is what you're using in your command, strace will actually call /usr/bin/printf version, not shell's version. Aside from minor differences, for our interest in this particular question the standard format specifiers are the same and behavior is the same.

What also should be kept in mind is that printf syntax is far more portable ( and therefore preferred ) than echo, not to mention that the syntax is more familiar to C or any C-like language that has printf() function in it. See this excellent answer by terdon on the subject of printf vs echo. While you can make the output tailored to your specific shell on your specific version of Ubuntu, if you are going to be porting scripts across different systems, you probably should prefer printf rather than echo. Maybe you're a beginner system administrator working with Ubuntu and CentOS machines, or maybe even FreeBSD - who knows - so in such cases you will have to make choices.

Quote from bash manual, SHELL BUILTIN COMMANDS section

read [-ers] [-a aname] [-d delim] [-i text] [-n nchars] [-N nchars] [-p prompt] [-t timeout] [-u fd] [name ...]

One line is read from the standard input, or from the file descriptor fd supplied as an argument to the -u option, and the first word is assigned to the first name, the second word to the second name, and so on, with leftover words and their intervening separa‐ tors assigned to the last name. If there are fewer words read from the input stream than names, the remaining names are assigned empty values. The characters in IFS are used to split the line into words using the same rules the shell uses for expansion (described above under Word Splitting).

Solution 2

This is only a suggestion and not meant to replace Sergiy's answer at all. I think Sergiy wrote a great answer as to why they are different in printing. How the variable on the read gets assigned with the remaining into the $c variable as 3 4 5 6 after 1 and 2 are assigned to a and b. echo won't split up the variable for you where printf will with the %ds.

You can, however, make them basically give you the same answers by manipulating the echo of the numbers at the beginning of the command:

In /bin/bash you can use:

echo -e "1 2 3 \n4 5 6"

In /bin/sh you can just use:

echo "1 2 3 \n4 5 6"

Bash uses the -e to enable the \ escape characters where sh does not need it as it is already enabled. \n causes it to create a new line, so now the echo line is split into two separate lines that can now be used two times for your echo loop statement:

:~$ echo -e "1 2 3 \n4 5 6" | while read a b c; do echo "$a, $b, $c"; done
1, 2, 3
4, 5, 6

Which in turn produces the same output with using the printf command:

:~$ echo -e "1 2 3 \n4 5 6" | while read a b c ;do printf "%d, %d, %d \n" $a $b $c; done
1, 2, 3 
4, 5, 6 

---

In sh

$ echo "1 2 3 \n4 5 6" | while read a b c; do echo "$a, $b, $c"; done
1, 2, 3
4, 5, 6
$ echo "1 2 3 \n4 5 6" | while read a b c ;do printf "%d, %d, %d \n" $a $b $c; done
1, 2, 3 
4, 5, 6 

Hope this helps!

Comments

[Charles Duffy]

echo -e is not just an extension to POSIX -- but any echo that doesn't emit -e on its output given that input is actually defying/breaking black-letter specification. (bash is noncompliant by default, but complies with the standard when both posix and xpg_echo flags are set; the latter can be made default at compile-time, meaning that not all versions of bash will work with echo -e as you expect here).

[Charles Duffy]

See APPLICATION USAGE and RATIONALE sections of the POSIX spec for echo at pubs.opengroup.org/onlinepubs/9699919799/utilities/echo.html‌​, explicitly describing that echo's behavior is unspecified in the presence of either -n or backslashes anywhere in input, and advising that printf be used instead.

[Terrance]

@CharlesDuffy Did I ever write in mine that I expect this to work in all versions of bash? And what problem do you have with my answer? If you feel you have a better solution, write it as an answer instead of trying to prove people wrong. I have been through this way too many times with people like you, so please stop with these comments like this. That's all I have to say.

[Charles Duffy]

The problem is precisely what I described above -- that you're encouraging (via demonstration) needlessly nonportable practices (that won't work in scripts using #!/bin/sh on Ubuntu). This site is a teaching resource, and what we teach is important. Editing to use printf '%s\n' "1 2 3" "4 5 6" or printf '%b' "1 2 3 \n4 5 6" in place of echo -e "1 2 3 \n4 5 6" would resolve the issue.

[Charles Duffy]

If you're going to write something that only works in some versions of bash, you should warn your readers so they don't try to use what they saw in your answer. If you don't provide a warning, it's generally reasonable for someone to expect to be able to reuse code they saw on a website that exists (after all) to teach them how to write code correctly.

[Charles Duffy]

One noteworthy difference between the strace case and the other -- strace printf is using /usr/bin/printf, whereas printf directly in bash is using the shell builtin by the same name. They won't always be identical -- for instance, the bash instance has format specifiers %q and, in new versions, $()T for time formatting.

[Charles Duffy]

(And to address your question/suggestion: I'm not adding my own answer because Sergiy's says pretty much everything I would need to. There's thus no need for more answers here, but I did feel it would be polite to comment on this one suggesting improvements to edit in before going straight to a downvote for the reasons above. Should said improvements be made, I'll of course be happy to withdraw same).

[Terrance]

@CharlesDuffy By the way, sh does not equal bash. Dash and Bash are two totally different shells. Ubuntu defaults to BASH. Secondly, /bin/sh would only need the -e removed from the echo line to produce the exact same results.

[WinEunuuchs2Unix]

@CharlesDuffy in Ask Ubuntu we sometimes write answers that are non-transportable to other Linux distros. As your rep here is only 103 points you might not have noticed that. Your rep on Stack Overflow though is 123.3K points so you obviously know a ton of stuff about *NIX systems in general. I think You, Terrace and Serg are all right but are looking at the thread from different angles. I echo (no pun intended) the sentiment for you to write an answer that is *NIX portable, or at least portable across Linux distros.

[Charles Duffy]

Some implementations of /bin/sh -- those which implement optional XSI extensions -- not all of them. Whereas if you used printf instead of echo, you'd have the same code working everywhere.

[Sergiy Kolodyazhnyy]

@CharlesDuffy While I agree with your sentiment answers should be made portable, this is after all, Ubuntu-oriented site, hence the Ubuntu-specific tools should be assumed by the users. So the warning is somewhat implied. Let's not go into overly lengthy discussions. You're right that other shells should be considered, and newbies reading to learn from answers should be considered as well. A short comment would probably be sufficient to make the point.