Create an associative array from the output of two commands

Question

I am trying to create user directories based on an imported passwd file, and am trying to load the data into an associative array: array[username]=directory . I can load the fields into a separate arrays but I cannot get the associations correct, as each field gets associated with all of the directories.

USERLIST=$(cat /usrmkr/in.out | awk -F ":" '{print $1}')
DIRLIST=$(cat /usrmkr/in.out | awk -F ":" '{print $6}')
declare -A USERARRAY

func_StoreData()
{
    USERARRAY[$1]="$2"
    return $?
}

for ((u=0;u<${USERLIST[@]};u++)); do
    func_StoreData ${USERLIST[$u]} ${DIRLIST[$u]}
done

for i in ${!USERARRAY[@]}; do
    echo "making directory for $i in ${USERARRAY[$i]}"
    #Do stuff
done

Solution 1

You could read the whole thing with bash directly, telling read to split on colons:

declare -A userarray
while IFS=: read -r username password uid gid gecos home shell; do
  userarray[$username]=$home
done < /usrmkr/in.out

Solution 2

eval declare -A USERARRAY=(
    $(awk -F: '{ printf "[\"%s\"]=\"%s\"\n", $1, $6}' /usrmkr/in.out)
)

The awk script produces output in the [key]=val format required when setting multiple elements of a bash associative array, and double-quotes both the key and the value (["key"]="value") in case there are spaces, tabs etc in either the keys or the values.

I've used \n as separator to make it easy in case you want to post-process the awk output with some other tool (although awk can do most things you might want to do with other tools, anyway).

Command substitution should be enough by itself...but due to what is, IMO, a bug in bash, if the first non-whitespace character inside the ( ... ) array definition isn't a [, it just produces the error message must use subscript when assigning associative array.

e.g. both of the following attempts to set USERARRAY will fail:

$ bash --version | head -1
GNU bash, version 4.3.46(1)-release (x86_64-pc-linux-gnu)

$ declare -A USERARRAY=($(awk -F: '{ printf "[\"%s\"]=\"%s\"\n", $1, $6}' /usrmkr/in.out))
bash: USERARRAY: $(awk -F: '{ printf "[\"%s\"]=\"%s\"\n", $1, $6}' /usrmkr/in.out): must use subscript when assigning associative array

$ UA=$(awk -F: '{ printf "[\"%s\"]=\"%s\"\n", $1, $6}' /usrmkr/in.out)
$ declare -A USERARRAY=( $UA )
bash: USERARRAY: $UA: must use subscript when assigning associative array

The solution is to use eval when declaring the hashed array, as in the code example at the top of my answer. Or,

eval declare -A USERARRY=( $UA )

Solution 3

Rather than merging two lists, we can build the array in a single loop (and a single call to awk for good measure) by return a list of user:dir entries then splitting that up with variable expansions:

#!/bin/bash

declare -A USERARRAY

for u in $(awk -F: '{print $1 ":" $6}' /usrmkr/in.out)
do
  user=${u%:*}
  dir=${u#*:}
  USERARRAY[$user]=$dir
done

Comments

[Jeff Schaller]

If any of the existing answers solves your problem, please consider accepting it via the checkmark. Thank you!

[rubystallion]

Why did you put '\n' twice into the IFS?

[Stéphane Chazelas]

@rubystallion see edit

[Alessio]

1. this uses eval - of course there's a risk of command injection. I don't think it's a problem in this case, though, because my answer is intended for use with the OP's specific & presumably vetted/known-good input file. 2. "forgot to quote the expansions" - huh? what? I don't see any quoting problem - the awk code explicitly quotes both keys and vals for the array. 3. BTW, nearly 2 years later, bash v4.4.19(1)-release still doesn't let you directly use a cmd-subst or even a var in a hash declaration: declare -A array=( $(...) ) fails with same error, so eval is still required

[Alessio]

@StéphaneChazelas If there's a quoting problem then please explain how and why - preferably with a real-world example using valid /etc/passwd style input, as is relevant to this question and answer, not just a generic and obvious "if you feed in garbage data, you'll get unpredictable and potentially dangerous results".

[Stéphane Chazelas]

You've got a point that it would typically not be a problem for typical /etc/passwd data. So let me rephrase it: ⚠ while it should be OK for typical /etc/passwd data, that approach should not be used for arbitrary data as it would constitute a command injection vulnerability. Specifically, it assumes the data doesn't contain, `, ", backslash, $, *, ?, [, space tab or NUL characters.

[Stéphane Chazelas]

About the quoting problem, see for instance Security implications of forgetting to quote a variable in bash/POSIX shells and the questions linked there to see what it means to leave expansions unquoted.

[Stéphane Chazelas]

Note that older versions of bash had a bug where 'a=("$x")` would fail to create an array with one element being the content of $x when $x was something like [123]=qwe (that was before bash supported associative arrays). It's true bash could have chosen to allow associative arrays to be declared as a=("$x") (where $x has to start with [...]=) in addition to the a=([$k]=$v) syntax but then, it would have had to define a way to specify keys that contain ] characters. In any case, I would not call it a bug but a design decision (here made by ksh where bash copied its syntax from).

[Alessio]

1. So, then, just the generic and obvious quoting issues that aren't actually relevant to this particular question with this particular set of input data. 2. I didn't forget to double-quote the $() or $UA variables. That was deliberate - it would be worse than pointless to treat either of them as one big string, at best that would result in an array with one element. Definitely not what is wanted here. 3. RE: defining arrays with keys containing ] etc - standard shell escaping and quoting seems to work as expected: e.g. typeset -A array=( ["[foo]"]="bar"); typeset -p array

[Stéphane Chazelas]

1,2. No, here you're using eval, you want that output to be passed as one single argument to eval. Invoking split+glob on it so that it be treated as an IFS-separated list of glob patterns resulting in a file list that eval joins with space before interpreting as shell code would not make sense whatsoever. 3. Quoting is shell code, surely you're not suggesting that in a=("$x"), the expansion of $x should undergo another round of shell code interpretation as if eval was called.

[Alessio]

1. Your concern is only valid if you assume the input data was random garbage. My answer, as I have said before, assumes that it is not - that the OP knew what he was doing and had vetted and/or fixed the input data so that it was known to be good. 2. I am suggesting that bash should do the right thing - i.e. do what is expected, without fuss, without throwing up stupid error messages and without requiring ugly workarounds like eval - and it should do that no matter what it takes, even if that means <gasp!> an extra round of expansion inside array definitions (i.e. treat like a subshell).