Why does this 'while' loop not recognize the last line?

Question

We use the following script:

more test.sh
#!/bin/bash

while read -r line
do

echo $line

done < /tmp/file

This is the file:

kafka-broker,log.retention.hours,12
kafka-broker,default.replication.factor,2
fefolp-defaults,fefolp.history.fs.cleaner.interval,1d
fefolp-defaults,fefolp.history.fs.cleaner.maxAge,2d
fefolp-env,fefolp_daemon_memory,10000
blo-site,blo.nodemanager.localizer.cache.target-size-mb,10240
blo-site,blo.nodemanager.localizer.cache.cleanup.interval-ms,300000
ams-env,metrics_collector_heapsize,512
fefolp,hbase_master_heapsize,1408
fefolp,hbase_regionserver_heapsize,512
fefolp,hbase_master_xmn_size,192
core-site,blolp.proxyuser.ambari.hosts,*
core-site,Hadoop.proxyuser.root.groups,*
core-site,Hadoop.proxyuser.root.hosts,*
blo-site,blo.scheduler.minimum-allocation-mb,1024
blolp-env,fefolp_heapsize,4096

Remark - after the last line - there are no space!

But the script prints only these lines (except the last line):

./test.sh
kafka-broker,log.retention.hours,12
kafka-broker,default.replication.factor,2
fefolp-defaults,fefolp.history.fs.cleaner.interval,1d
fefolp-defaults,fefolp.history.fs.cleaner.maxAge,2d
fefolp-env,fefolp_daemon_memory,10000
blo-site,blo.nodemanager.localizer.cache.target-size-mb,140
blo-site,blo.nodemanager.localizer.cache.cleanup.interval-ms,300
ams-env,metrics_collector_heapsize,51
fefolp,hbase_master_heapsize,1408
fefolp,hbase_regionserver_heapsize,542
fefolp,hbase_master_xmn_size,19
core-site,blolp.proxyuser.ambari.hosts,*
core-site,Hadoop.proxyuser.root.groups,*
core-site,Hadoop.proxyuser.root.hosts,*
blo-site,blo.scheduler.minimum-allocation-mb,1024

Why does this happen?

Solution

Your input text contains an incomplete line as its last line. The last line is not terminated by a newline.

while IFS= read -r line || [ -n "$line" ]; do
    printf '%s\n' "$line"
done <file

The above loop will read unmodified lines¹ (without stripping whitespaces or interpreting backslashed control sequences) from the file called file and print them to standard output.

When an incomplete line is read, read will fail, but $line will still contain data. The extra -n test will detect this so that the loop body is allowed to output the incomplete line. In the iteration after that, read will fail again and $line will be an empty string, thus terminating the loop.

---

^{¹ assuming they don't contain NUL characters in shells other than zsh and assuming they don't contain sequences of bytes not forming part of valid characters in the yash shell, both of which shouldn't happen if the input is valid text, though that missing line delimiter on the last line already makes it invalid text.}

Comments

[Rui F Ribeiro]

because the last text line is not complete. you need a \n there.

[yael]

do you have suggestion how to add \n on the last line in the file , so I will put thus in the scritp

[Rui F Ribeiro]

echo >> file as a poor mans solution.

[Kamil Maciorowski]

Note POSIX defines "line" as a sequence of zero or more non- <newline> characters plus a terminating <newline> character. Your last line isn't even a line in this context.

[AdminBee]

Does this answer your question? Read a line-oriented file which may not end with a newline

[Kusalananda]

@yael You say in the question that you use a loop like the one you are showing. I gave a solution for how to fix that loop. It does not matter what the body of the loop looks like, $line will be each individual line of the file, including the last incomplete one. If you don't use a loop like this, what do you use?

[yael]

ok , so maybe I will post another question for how to add "\n" on the last line

[Kusalananda]

@yael How to add a newline to the end of a file?

[bot47]

@Kusalananda Could you please also answer this one with a nearly identical answer? superuser.com/questions/1500525/reason-for-behind-read

[muru]

Just do { cat /tmp/file; echo; } if you're going to do that: unix.stackexchange.com/a/418081/70524

[Nathan SR]

We could use any outputting program instead of cat /tmp/file hence the given structure / syntax.

[muru]

Yes, you could use any program: { foo ; echo; } | ...