shell - temp IFS as newline only. Why doesn't this work: IFS=$(echo -e '\n')

Question

I'm trying to use for in the shell to iterate over filenames with spaces. I read in a stackoverflow question that IFS=$'\n' could be used and that seems to work fine. I then read in a comment to one of the answers that to make it posix compatible for shells other than bash one could use IFS=$(echo -e '\n').

The former works but the latter doesn't. The comment is upvoted several times. I checked the output and the variable doesn't seem to contain the newline.

#!/bin/sh

IFS=$(echo -e '\n')
echo -n "$IFS" | od -t x1
for file in `printf 'one\ntwo two\nthree'`; do
    echo "Found $file"
done

echo

IFS=$'\n'
echo -n "$IFS" | od -t x1
for file in `printf 'one\ntwo two\nthree'`; do
    echo "Found $file"
done

The output shows the field separator is missing for the first:

0000000
Found one
two two
three

But is proper on the second:

0000000 0a
0000001
Found one
Found two two
Found three

I found an answered question which may or may not be a duplicate of my question:
linux - When setting IFS to split on newlines, why is it necessary to include a backspace? - Stack Overflow

Is there any danger to doing what's discussed there, IFS=$(echo -en '\n\b')? Because that adds a \b to IFS (I checked). Can a \b occur in a file name? I don't want my code to split a file name by mistake.

Also do you have any recommendations on how to properly handle restoration of IFS after the for loop? Should I store it in a temporary variable, or run IFS and the for statement in a subshell? (and how to do that?) Thanks

Solution 1

Update - changing my pseudo-comment to a real answer.

I think this answer should explain the behavior you are seeing. Specifically command substitution operators $() and backticks will strip trailing newlines from the command output. However the direct assignment in your second example doesn't do any command subsitution, so works as expected.

So I'm afraid to say I think the upvoted comment you refer to is incorrect.

---

I think the safest way to restore IFS is to set it in a subshell. All you need to do is put the relevant commands in parentheses ():

(
    IFS=$'\n'
    echo -n "$IFS" | od -t x1
    for file in `printf 'one\ntwo two\nthree'`; do
        echo "Found $file"
    done
)

Of course invoking a subshell incurs a small delay, so performance needs to be considered if this is to be repeated many times.

---

As an aside, be very careful, filenames can contain both \b and \n. I think just about the only characters they cannot contain are slash and \0. At least thats what it says on this wikipedia article.

$ touch $'12345\b67890'
$ touch "new
> line"
$ ls --show-control-chars
123467890  new
line
$ 

Solution 2

Since newlines are stripped by command substitution, as @DigitalTrauma correctly wrote, you should use a different, POSIX compliant way.

IFS='
'

writes a newline and assign it to it. Simple and effective.

If you don't like assignments that span over two lines, you may use printf as alternative.

IFS="$(printf '\n')"

Solution 3

IFS=$(printf '\n+'); IFS=${IFS%?}

As mentioned by others, the command substitution removes trailing newlines. So, if you want something "readable" then add a character after the newline ('+' in this case) to prevent it from being removed and then use a parameter expansion to remove the placeholder character.

Comments

[user2672807]

Thank you for the information I will keep that in mind

[Tom Hale]

Please remove your 2nd example as it isn't clean: IFS="$(printf '\n+')" $ echo -n "$IFS" | xxd 00000000: 0a2b <- This adds + to IFS also.

[Riccardo Galli]

@TomHale I've removed the + which wasn't needed

[Tom Hale]

Did you test it? That doesn't work for the reason given here.

[Riccardo Galli]

I did, it works. Btw I'm testing on rxvt-unicode on Ubuntu (so dash I suppose for /bin/sh)