bash array with variable in the name

Question

I'll appreciate your help with the following issue:

I'm trying to set an array which contains a variable as part of the array name, example: Arr_$COUNTER (where $COUNTER is changed based on a loop count)

Every possible way I have tried came up with an error, such as "bad substitution" or "syntax error near unexpected token"

Here is the entire flow:

1. There is one file which contain multiple lines. each line has 6 values separated by space

   10 20 30 40 50 60  
   100 200 300 400 500 600
   

2. The script, is meant to read each line from the file, and declare it as an array (with the line number which is the variable.

3. as a test, each value should be printed and eventually another function will be executed on each value.

   #!/bin/bash
   COUNTER=1
   LINES=`wc -l VALUES_FILE.txt | awk '{print $1}'`
   echo "Total number of lines "$LINES
   echo
   while [ $COUNTER -le $LINES ]
   do
   echo "Counter value is $COUNTER"
   field=`awk "NR == $COUNTER" VALUES_FILE.txt`
   echo "Field = $field"
   declare -a "arr$COUNTER=($field)"
   echo "arr$COUNTER[0] = ${arr$COUNTER[0]}"
   echo "arr$COUNTER[1] = ${arr$COUNTER[1]}"
   echo "arr$COUNTER[2] = ${arr$COUNTER[2]}"
   echo "arr$COUNTER[3] = ${arr$COUNTER[3]}"
   echo "arr$COUNTER[4] = ${arr$COUNTER[4]}"
   echo "arr$COUNTER[5] = ${arr$COUNTER[5]}"
   let COUNTER=COUNTER+1
   echo
   done
   echo "The End"
   echo
   

Here is the result:

Total number of lines 2

Counter value is 1
Field = 10 20 30 40 50 60
./sort.sh: line 12: arr$COUNTER[0] = ${arr$COUNTER[0]}: bad substitution
The End

What should be changed / fixed in order to have it working properly?

thank !

Solution 1

Some ideas:

1. A "parameter expansion" of a variable value (the ${...} part):

   echo "arr$COUNTER[0] = ${arr$COUNTER[0]}"
   

   will not work. You may get around by using eval (but I do not recommend it):

   eval echo "arr$COUNTER[0] = \${arr$COUNTER[0]}"
   

   That line could be written as this:

   i="arr$COUNTER[0]"; echo "$i = ${!i}"
   

   That is called indirection (the !) in Bash.

2. A similar issue happens with this line:

   declare -a "arr$COUNTER=($field)"
   

   Which should be split into two lines, and eval used:

   declare -a "arr$COUNTER"
   eval arr$COUNTER\=\( \$field \)
   

   Again, I do not recommend using eval (in this case).

3. As you are reading the whole file into the memory of the shell, we may as well use a simpler method to get all lines into an array:

   readarray -t lines <"VALUES_FILE.txt"
   

   That should be faster than calling awk for each line.

An script with all the above could be:

#!/bin/bash
valfile="VALUES_FILE.txt"

readarray -t lines <"$valfile"             ### read all lines in.

line_count="${#lines[@]}"
echo "Total number of lines $line_count"

for ((l=0;l<$line_count;l++)); do
    echo "Counter value is $l"             ### In which line are we?
    echo "Field = ${lines[l]}"             ### kepth only to help understanding.
    k="arr$l"                              ### Build the variable arr$COUNTER
    IFS=" " read -ra $k <<<"${lines[l]}"   ### Split into an array a line.
    eval max=\${#$k[@]}                    ### How many elements arr$COUNTER has?
    #echo "field $field and k=$k max=$max" ### Un-quote to "see" inside.
    for ((j=0;j<$max;j++)); do             ### for each element in the line.
        i="$k[$j]"; echo "$i = ${!i}"      ### echo it's value.
    done
done
echo "The End"
echo

However, still, AWK may be faster, if we could execute what you need in AWK.

---

A similar processing could be done in awk. Assuming the 6 values will be used as an IP (4 of them) and the other two are a number and an epoch time.

Just a very simple sample of an AWK script:

#!/bin/sh
valfile="VALUES_FILE.txt"
awk '
NF==6 { printf ( "IP: %s.%s.%s.%s\t",$1,$2,$3,$4)
        printf ( "number: %s\t",$5+2)
        printf ( "epoch: %s\t",$6)
        printf ( "\n" )
    }
' "$valfile"

Just make a new question with the details.

Solution 2

You can use the variable indirection, if you assign both the name and index to a variable:

s="arr$COUNTER[0]"
echo "arr$COUNTER[0] = ${!s}"

Solution 3

You can generate names using eval, e.g.,

eval declare -a '"arr'$COUNTER'=($field)"'

essentially quoting all of the meta-characters except the ones you want to evaluate.

So... if $COUNTER is 1, your script would do

declare -a "arr1=($field)"

Further reading:

• eval command in Bash and its typical uses
• What is the “eval” command in bash?

Solution 4

The standard way of storing multi-dimensional array data in an array of dimension 1 is to store each row at an offset into the array.

Element (i,j) will be located at index i*m + j where i is the zero-based row index, j is the zero-based column index, and m is the number of columns.

This also makes it easier to read the data in as we can just take your input file, change all spaces to newlines and use readarray.

On the command line:

$ readarray -t arr < <( tr -s ' ' '\n' <data )
$ printf '%s\n' "${arr[@]}"
10
20
30
40
50
60
100
200
300
400
500
600

We can figure out the number of columns in the data with

$ m=$( awk '{ print NF; exit }' <data )

And the number of rows:

$ n=$( wc -l <data )

Then we may iterate over columns and rows in a double loop as usual:

for (( i = 0; i < n; ++i )); do
    for (( j = 0; j < m; ++j )); do
        printf '%4d' "${arr[i*m + j]}"
    done
    printf '\n'
done

For the given data, this would generate

  10  20  30  40  50  60
 100 200 300 400 500 600

---

Ideally, you'd use a language, such as Perl or Python or C, that supports multi-dimensional arrays. That is, if you actually need to store the whole set of data in memory at all and can't process it on a row by row basis.

For row by row processing, awk would be a good candidate to replace bash (any language would be a good candidate to replace a shell for any sort of data processing):

awk '{ for (i = 1; i <= NF; ++i) printf("%4d", $i); printf("\n") }' data

View more solutions

Comments

[AlonCo]

This might work... I'll test it and update. Thanks?!

[AlonCo]

I'm not sure I get it. Can you explain further?

[Marius]

eval is POSIX (works "everywhere"), but keep in mind that all of the generated names would get similar treatment. Readability of the script is a problem with either eval or variable indirection.

[choroba]

Variable indirection ${!s} works with arrays only if you use both the array name and index. See "variable indirection" under "Parameter Expansion" in man bash.

[AlonCo]

Thank you for the detailed information - that helped ! unfortunately the "readarray" is not working for me ..

[Admin]

@AlonCo Assuming readarray is not available because of your bash version, then take a look to the detailed analysis in here. I am sure you will find a solution there.

[AlonCo]

my script is completed thanks to you and as you already guessed it's SLOW. You mentioned AWK can be faster, how? The script eventually will be used to read a large file ( 100k lines) where each line has 6 values and convert these values to IP / number/ epoch time.

[Admin]

@AlonCo I added an AWK script to only print the values to my answer. Take a look, time it. It does not read the whole file into memory (which should be an additional plus IMO).

[AlonCo]

thank you again ! the script was so slow I tried using awk as you suggested. I'm happy to say awk is MUCH faster and funny... the script ended up much smaller as well :-) . thanks again !

[Admin]

@AlonCo You are welcome, I am happy to know it helped, enjoy!.

[opinion_no9]

works fine, actually. I use something like <br/> declare -a ${gral}[2]="cow" <br/> and the assigned value can just be a variable. The index can be a variable, too (depends on associated or indexed type). You may have a look here, they do that stuff as well: shell-tips.com/bash/arrays