read -a array -d '\n' < foo, exit code 1

Question

If I try to execute

read -a fooArr -d '\n' < bar

the exit code is 1 -- even though it accomplishes what I want it to; put each line of bar in an element of the array fooArr (using bash 4.2.37).

Can someone explain why this is happening

---

I've found other ways to solve this, like the ones below, so that's not what I'm asking for.

for ((i=1;; i++)); do
    read "fooArr$i" || break;
done < bar

or

mapfile -t fooArr < bar

Solution 1

What needs to be explained is that the command appeared to work, not its exit code

'\n' is two characters: a backslash \ and a letter n. What you thought you needed was $'\n', which is a linefeed (but that wouldn't be right either, see below).

The -d option does this:

  -d delim  continue until the first character of DELIM is read, rather
            than newline

So without that option, read would read up to a newline, split the line into words using the characters in $IFS as separators, and put the words into the array. If you specified -d $'\n', setting the line delimiter to a newline, it would do exactly the same thing. Setting -d '\n' means that it will read up to the first backslash (but, once again, see below), which is the first character in delim. Since there is no backslash in your file, the read will terminate at the end of file, and:

Exit Status:
The return code is zero, unless end-of-file is encountered, read times out,
or an invalid file descriptor is supplied as the argument to -u.

So that's why the exit code is 1.

From the fact that you believe that the command worked, we can conclude that there are no spaces in the file, so that read, after reading the entire file in the futile hope of finding a backslash, will split it by whitespace (the default value of $IFS), including newlines. So each line (or each word, if a line contains more than one word) gets stashed into the array.

The mysterious case of the purloined backslash

Now, how did I know the file didn't contain any backslashes? Because you didn't supply the -r flag to read:

  -r                do not allow backslashes to escape any characters

So if you had any backslashes in the file, they would have been stripped, unless you had two of them in a row. And, of course, there is the evidence that read had an exit code of 1, which demonstrates that it didn't find a backslash, so there weren't two of them in a row either.

Takeaways

Bash wouldn't be bash if there weren't gotchas hiding behind just about every command, and read is no exception. Here are a couple:

1. Unless you specify -r, read will interpret backslash escape sequences. Unless that's actually what you want (which it occasionally is, but only occasionally), you should remember to specify -r to avoid having characters disappear in the rare case that there are backslashes in the input.

2. The fact that read returns an exit code of 1 does not mean that it failed. It may well have succeeded, except for finding the line terminator. So be careful with a loop like this: while read -r LINE; do something with LINE; done because it will fail to do something with the last line in the rare case that the last line doesn't have a newline at the end.

3. read -r LINE preserves backslashes, but it doesn't preserve leading or trailing whitespace.

Solution 2

That is the expected behaviour:

The return code is zero, unless end-of-file is encountered, [...]

start cmd:> echo a b c | { read -a testarray; echo $?; }
0

start cmd:> echo -n a b c | { read -a testarray; echo $?; }
1

Comments

[Angel Todorov]

"the rare case that the last line didn't have a newline" does not seem to me to be a compelling reason to advise "never use while read". Otherwise, fantastic answer.

[rici]

@glennjackman: You can use while read as long as you don't have anything inside the loop. Otherwise, it's a bug waiting to bite you -- and believe me, I've been bitten.

[Angel Todorov]

That makes no sense to me. How do you iterate through a file?

[Gilles 'SO- stop being evil']

The last line always has a newline. That's how a line is defined: it ends with a newline. If a non-empty file doesn't end with a newline, it isn't a text file, and you can't use text processing tools such as the shell utility read.

[rici]

@glennjackman: Me, I iterate through columnised files with awk, mostly. For iterating whole lines where the file isn't too big, mapfile is pretty cool. For a quick&dirty hack, i'll use the forbidden while loop, but I've stopped putting that in production scripts. YMMV and I softened the warning in the answer.

[Angel Todorov]

Coming back with hindsight: while IFS= read -r line || [[ -n "$line" ]]; do echo "$line"; done < file handles files where the last line does not end with a newline.