Split command output by linebreak?

Question

I have a command returning multiple lines. For further processing I need to process each single line of those lines.

My current code works by modifying the IFS (Internal Field Separator):

ROWS=$(some command returning multiple lines)

O=$IFS #save original IFS
IFS=$(echo -en "\n\b") # set IFS to linebreak

for ROW in ${ROWS[@]}
do
  echo "$ROW"
done

IFS=$O #restore old IFS

I am wondering, is there a better way to access the single lines of the multiple lines output, one without modifying the IFS? Especially the readability of my script gets bad by modifying the IFS.

Update: I have troubles to get the answers working, e.g the one from choroba:

while IFS= read -r line ; do
    let var+=line #line 42
done << $(sqlite3 -list -nullvalue NULL -separator ',' /var/log/asterisk/master.db "${QUERY}")
echo "$var" # line 44

gives me

./bla.sh: row 44: Warning: here-document at line 43 delimited by end-of-file (wanted `$(sqlite3 -list -nullvalue NULL -separator , /var/log/asterisk/master.db ${QUERY})')
./bla.sh: row 42: let: echo "": syntax error: invalid arithmetic operator. (error causing character is \"""\").

Anyone can help me with this? Thanks!

Solution 1

What about read in a while loop?

some command returning multiple lines | while IFS= read -r line ; do
    echo "$line"
done

Be careful, though, the pipe is run in a subshell, which means you cannot change variable values for the rest of the script. If you need something to survive from your while loop, you can use bash's process substitution:

while IFS= read -r line ; do
    let var+=line
done < <(some command returning multiple lines)
echo "$var"

Solution 2

Setting IFS to only a newline is not enough. (Why are you also splitting at backspace characters, by the way?)

In your code, ${ROWS[@]} (which is a strange way to write $ROWS — ROWS is not an array) is not double-quoted. (If it was inside double quotes, you'd get a single string, since ROWS is not an array.) So the shell splits the value of the variable into fields at each IFS character, then treats each field as a glob pattern. For example, if one of the lines printed by the command contains the single character *, this will be replaced by the file names in the current directory.

You can turn off globbing with set -f. In most cases where you set IFS to use the shell's field splitting feature, you also need to turn off globbing. Set it back on with set +f.

The reliable idiom for reading a command's output line by line is while IFS= read -r.

some command returning multiple lines |
while IFS= read -r ROW; do
  …
done

Note that most shells run each command of a pipeline in a separate subshell. So if you need to set variables and use them after the loop, wrap these commands in a group together with the loop. (Ksh and zsh are the exceptions, they run the last command of a pipeline in the parent shell.)

some command returning multiple lines | {
  while IFS= read -r ROW; do
    …
    row_count=$((row_count+1))
  done
  echo "There were $row_count rows."
}

Solution 3

You are mixing the here-document and here-string syntax in your update question.

Either use here-document:

while IFS= read -r line ; do
    let var+=line #line 42
done <<ENDMARK
$(sqlite3 -list -nullvalue NULL -separator ',' /var/log/asterisk/master.db "${QUERY}")
ENDMARK

Or here-string:

while IFS= read -r line ; do
    let var+=line #line 42
done <<< $(sqlite3 -list -nullvalue NULL -separator ',' /var/log/asterisk/master.db "${QUERY}")

Comments

[Admin]

choroba's answer says to do < <(some command returning multiple lines), but that's not what you are doing.

[Admin]

question with +922 total votes: How do I split a string on a delimiter in Bash?

[tcoolspy]

Using process substitution is a smart solution for this in bash, but it's worth noting that the subshell scoping of variables is only a problem in bash, doing the same thing in zsh would not have this issue at all.

[Razzlero]

ksh doesn't have this issue either.

[Nik]

thank you guys, I tried this solution, but I can't get it working, please see my updated original post, thanks!

[choroba]

@stefan.at.wpf: You cannot place spaces and dollar signs around at will. They have meaning.