Multivariable For Loops

Question

Is there a way to specify multiple variables (not just integers) in for loops in bash? I may have 2 files containing arbitrary text that i would need to work with.

What i functionally need is something like this:

for i in $(cat file1) and j in $(cat file2); do command $i $j; done

Any ideas?

Solution 1

First, Don't read lines with for, as there are several unavoidable issues with reading lines via word-splitting.

Assuming files of equal length, or if you want to only loop until the shorter of two files are read, a simple solution is possible.

while read -r x && read -r y <&3; do
    ...
done <file1 3<file2

Putting together a more general solution is hard because of when read returns false and several other reasons. This example can read an arbitrary number of streams and return after either the shortest or longest input.

#!/usr/bin/env bash

# Open the given files and assign the resulting FDs to arrName.
# openFDs arrname file1 [file2 ...]
openFDs() {
    local x y i arr=$1

    [[ -v $arr ]] || return 1
    shift

    for x; do
        { exec {y}<"$x"; } 2>/dev/null || return 1
        printf -v "${arr}[i++]" %d "$y"
    done
}

# closeFDs FD1 [FD2 ...]
closeFDs() {
    local x
    for x; do
        exec {x}<&-
    done
}

# Read one line from each of the given FDs and assign the output to arrName.
# If the first argument is -l, returns false only when all FDs reach EOF.
# readN [ -l ] arrName FD1 [FD2 ...]
readN() {
    if [[ $1 == -l ]]; then
        local longest
        shift
    else
        local longest=
    fi

    local i x y status arr=$1
    [[ -v $arr ]] || return 1
    shift

    for x; do
        if IFS= read -ru "$x" "${arr}[i]" || { unset -v "${arr}[i]"; [[ ${longest+_} ]] && return 1; }; then
            status=0
        fi
        ((i++))
    done
    return ${status:-1}
}

# readLines file1 [file2 ...]
readLines() {
    local -a fds lines
    trap 'closeFDs "${fds[@]}"' RETURN
    openFDs fds "$@" || return 1

    while readN -l lines "${fds[@]}"; do
        printf '%-1s ' "${lines[@]}"
        echo
    done
}

{
    readLines /dev/fd/{3..6} || { echo 'error occured' >&2; exit 1; }
} <<<$'a\nb\nc\nd' 3<&0 <<<$'1\n2\n3\n4\n5' 4<&0 <<<$'x\ny\nz' 5<&0 <<<$'7\n8\n9\n10\n11\n12' 6<&0

# vim: set fenc=utf-8 ff=unix ts=4 sts=4 sw=4 ft=sh nowrap et:

So depending upon whether readN gets -l, the output is either

a 1 x 7
b 2 y 8
c 3 z 9
d 4 10
5 11
12

or

a 1 x 7
b 2 y 8
c 3 z 9

Having to read multiple streams in a loop without saving everything into multiple arrays isn't all that common. If you just want to read arrays you should have a look at mapfile.

Solution 2

Do-done; done - you need two fors and two dones:

for i in $(< file1); do for j in $(< file2); do echo $i $j;done ; done

File2 is, of course, processed for each word in file1 once.

Using < instead of cat should be an unremarkable performance gain in most cases. No subprocess involved. (Unsure about last sentence).

Uncompressed:

for i in $(< file1)
do
  for j in $(< file2)
  do
    echo $i $j
  done
done

If you don't like to read words, but lines, and preserve the whitespace, use while and read:

while read line; do while read second; do echo "${line}${second}"; done <file2 ; done <file1

If you want to append 2 files, and not nest them:

cat file1 file2 | while read line; do echo "${line}"; done

If you want to zip 2 files, use paste:

paste file1 file2

Comments

[Silverrocker]

You say there is no subprocess involed. Isn't the () a subshell? I don't want to be pendantic but I'm unsure what is happening now. For instance I know that when I have a script and I don't want it to polute the current shell with cd's for instance, I do it inside a (). Also running (sleep 4)& for example followed by a ps shows me the process is still running. Can you please elaborate.

[user unknown]

Well - I'm sorry, I'm very much acting without fundament here. I thought I heared so, but maybe it is only the < compared to cat. And I'm unsure how to test it, and when it will become semantically important. If pipes are invoked, variables in subprocesses don't bubble up to the main process, but we don't have variables here. I strike through the critical sentence, until somebody can clarify it.

[ormaaj]

I didn't have a free moment to test - yes you would probably want both elements per iteration. The || "list" operator is short-circuiting like in most languages. Surely this has been answered a thousand times before...

[Peter.O]

The point is not the frequency of how often something has been mentioned before. Rather, it is that the code you are currently showing in your answer does not work. Based on your " probably want both elements..", it sounds like you still haven't tested it.

[ormaaj]

@Peter.O I'm aware. It's been fixed.