Accessing a JSON object in Bash - associative array / list / another model

Question

I have a Bash script which gets data in JSON, I want to be able to convert the JSON into an accessible structure - array / list / or other model which would be easy to parse the nested data.

Example:

{
  "SALUTATION": "Hello world",
  "SOMETHING": "bla bla bla Mr. Freeman"
}

I want to get the value like the following: echo ${arr[SOMETHING]}

[ Different approach is optional as well. ]

Solution 1

If you want key and value, and based on How do i convert a json object to key=value format in JQ, you can do:

$ jq -r "to_entries|map(\"\(.key)=\(.value|tostring)\")|.[]" file
SALUTATION=Hello world
SOMETHING=bla bla bla Mr. Freeman

In a more general way, you can store the values into an array myarray[key] = value like this, just by providing jq to the while with the while ... do; ... done < <(command) syntax:

declare -A myarray
while IFS="=" read -r key value
do
    myarray[$key]="$value"
done < <(jq -r 'to_entries|map("(.key)=(.value)")|.[]' file)

And then you can loop through the values like this:

for key in "${!myarray[@]}"
do
    echo "$key = ${myarray[$key]}"
done

For this given input, it returns:

SALUTATION = Hello world
SOMETHING = bla bla bla Mr. Freeman

Solution 2

Although this question is answered, I wasn't able to fully satiate my requirements from the posted answer. Here is a little write up that'll help any bash-newcomers.

Foreknowledge

A basic associative array declaration

#!/bin/bash

declare -A associativeArray=([key1]=val1 [key2]=val2)

You can also use quotes (', ") around the declaration, its keys, and values.

#!/bin/bash

declare -A 'associativeArray=([key1]=val1 [key2]=val2)'

And you can delimit each [key]=value pair via space or newline.

#!/bin/bash

declare -A associativeArray([key1]=value1
  ['key2']=value2 [key3]='value3'
  ['key4']='value2'               ["key5"]="value3"


  ["key6"]='value4'
  ['key7']="value5"
)

Depending on your quote variation, you may need to escape your string.

Using Indirection to access both key and value in an associative array

example () {
  local -A associativeArray=([key1]=val1 [key2]=val2)

  # print associative array
  local key value
  for key in "${!associativeArray[@]}"; do
    value="${associativeArray["$key"]}"
    printf '%s = %s' "$key" "$value"
  done
}

Running the example function

$ example
key2 = val2
key1 = val1

Knowing the aforementioned tidbits allows you to derive the following snippets:

---

The following examples will all have the result as the example above

String evaluation

#!/usr/bin/env bash

example () {
  local arrayAsString='associativeArray=([key1]=val1 [key2]=val2)'
  local -A "$arrayAsString"

  # print associative array
}

Piping your JSON into JQ

#!/usr/bin/env bash
# Note: usage of single quotes instead of double quotes for the jq
#       filter. The former is preferred to avoid issues with shell 
#       substitution of quoted strings.
example () {
  # Given the following JSON
  local json='{ "key1": "val1", "key2": "val2" }'

  # filter using `map` && `reduce`
  local filter='to_entries | map("[\(.key)]=\(.value)") |
    reduce .[] as $item ("associativeArray=("; . + ($item|@sh) + " ") + ")"'

  # Declare and assign separately to avoid masking return values.
  local arrayAsString;
  # Note: no encompassing quotation (")
  arrayAsString=$(cat "$json" | jq --raw-output "${filter}")
  local -A "$arrayAsString"

  # print associative array
}

jq -n / --null-input option + --argfile && redirection

#!/usr/bin/env bash

example () {
  # /path/to/file.json contains the same json as the first two examples
  local filter filename='/path/to/file.json'

  # including bash variable name in reduction
  filter='to_entries | map("[\(.key | @sh)]=\(.value | @sh) ")
    | "associativeArray=(" + add + ")"'

  # using --argfile && --null-input
  local -A "$(jq --raw-output --null-input --argfile file "$filename" \
    "\$filename | ${filter}")"

  # or for a more traceable declaration (using shellcheck or other) this
  # variation moves the variable name outside of the string

  # map definition && reduce replacement
  filter='[to_entries[]|"["+(.key|@sh)+"]="+(.value|@sh)]|"("+join(" ")+")"'
  
  # input redirection && --join-output
  local -A associativeArray=$(jq --join-output "${filter}" < "${filename}")
  
  # print associative array
}

---

Reviewing previous answers

@Ján Lalinský

To load JSON object into a bash associative array efficiently (without using loops in bash), one can use tool 'jq', as follows.

# first, load the json text into a variable:
json='{"SALUTATION": "Hello world", "SOMETHING": "bla bla bla Mr. Freeman"}'

# then, prepare associative array, I use 'aa':
unset aa
declare -A aa

# use jq to produce text defining name:value pairs in the bash format
# using @sh to properly escape the values
aacontent=$(jq -r '. | to_entries | .[] | "[\"" + .key + "\"]=" + (.value | @sh)' <<< "$json")

# string containing whole definition of aa in bash
aadef="aa=($aacontent)"

# load the definition (because values may contain LF characters, aadef must be in double quotes)
eval "$aadef"

# now we can access the values like this: echo "${aa[SOMETHING]}"

Warning: this uses eval, which is dangerous if the json input is from unknown source (may contain malicious shell commands that eval may execute).

This could be reduced to the following

example () {
  local json='{ "key1": "val1", "key2": "val2" }'
  local -A associativeArray="($(jq -r '. | to_entries | .[] |
    "[\"" + .key + "\"]=" + (.value | @sh)' <<< "$json"))"

  # print associative array
}

@fedorqui

If you want key and value, and based on How do i convert a json object to key=value format in JQ, you can do:

$ jq -r "to_entries|map(\"\(.key)=\(.value|tostring)\")|.[]" file
SALUTATION=Hello world
SOMETHING=bla bla bla Mr. Freeman

In a more general way, you can store the values into an array myarray[key] = value like this, just by providing jq to the while with the while ... do; ... done < <(command) syntax:

declare -A myarray
while IFS="=" read -r key value
do
    myarray[$key]="$value"
done < <(jq -r "to_entries|map(\"\(.key)=\(.value)\")|.[]" file)

And then you can loop through the values like this:

for key in "${!myarray[@]}"
do
    echo "$key = ${myarray[$key]}"
done

For this given input, it returns:

SALUTATION = Hello world
SOMETHING = bla bla bla Mr. Freeman

The main difference between this solution and my own is looping through the array in bash or in jq.

Each solution is valid and depending on your use case, one may be more useful then the other.

Solution 3

_{Context: This answer was written to be responsive to a question title which no longer exists.}.

---

The OP's question actually describes objects, vs arrays.

To be sure that we help other people coming in who are actually looking for help with JSON arrays, though, it's worth covering them explicitly.

---

For the safe-ish case where strings can't contain newlines (and when bash 4.0 or newer is in use), this works:

str='["Hello world", "bla bla bla Mr. Freeman"]'
readarray -t array <<<"$(jq -r '.[]' <<<"$str")"

To support older versions of bash, and strings with newlines, we get a bit fancier, using a NUL-delimited stream to read from jq:

str='["Hello world", "bla bla bla Mr. Freeman", "this is\ntwo lines"]'
array=( )
while IFS= read -r -d '' line; do
  array+=( "$line" )
done < <(jq -j '.[] | (. + "\u0000")')

Solution 4

This is how can it be done recursively:

#!/bin/bash

SOURCE="$PWD"
SETTINGS_FILE="$SOURCE/settings.json"
SETTINGS_JSON=`cat "$SETTINGS_FILE"`

declare -A SETTINGS

function get_settings() {
    local PARAMS="$#"
    local JSON=`jq -r "to_entries|map(\"\(.key)=\(.value|tostring)\")|.[]" <<< "$1"`
    local KEYS=''

    if [ $# -gt 1 ]; then
        KEYS="$2"
    fi

    while read -r PAIR; do
        local KEY=''

        if [ -z "$PAIR" ]; then
            break
        fi

        IFS== read PAIR_KEY PAIR_VALUE <<< "$PAIR"

        if [ -z "$KEYS" ]; then
            KEY="$PAIR_KEY"
        else
            KEY="$KEYS:$PAIR_KEY"
        fi

        if jq -e . >/dev/null 2>&1 <<< "$PAIR_VALUE"; then
            get_settings "$PAIR_VALUE" "$KEY"
        else
            SETTINGS["$KEY"]="$PAIR_VALUE"
        fi
    done <<< "$JSON"
}

To call it:

get_settings "$SETTINGS_JSON"

The array will be accessed like so:

${SETTINGS[grandparent:parent:child]}

Solution 5

To load JSON object into a bash associative array efficiently (without using loops in bash), one can use tool 'jq', as follows.

# first, load the json text into a variable:
json='{"SALUTATION": "Hello world", "SOMETHING": "bla bla bla Mr. Freeman"}'

# then, prepare associative array, I use 'aa':
unset aa
declare -A aa

# use jq to produce text defining name:value pairs in the bash format
# using @sh to properly escape the values
aacontent=$(jq -r '. | to_entries | .[] | "[\"" + .key + "\"]=" + (.value | @sh)' <<< "$json")

# string containing whole definition of aa in bash
aadef="aa=($aacontent)"

# load the definition (because values may contain LF characters, aadef must be in double quotes)
eval "$aadef"

# now we can access the values like this: echo "${aa[SOMETHING]}"

Warning: this uses eval, which is dangerous if the json input is from unknown source (may contain malicious shell commands that eval may execute).

View more solutions

Comments

[logicbloke]

It just took the root element and added it to the array. It's not recursive.

[HelpNeeder]

@PhoenixNoor, please look at my answer for recursive way: stackoverflow.com/a/47026579/720323

[Charles Duffy]

As an aside, function foo() { combines two separate function declaration syntax forms -- the ksh-compatible function foo {, and the POSIX-compatible foo() { -- while itself being compatible with neither (and not supporting the variables-local-by-default behavior that function added in old ksh). Consider picking one or the other; see wiki.bash-hackers.org/scripting/obsolete for further background.

[Charles Duffy]

Another option, btw, is to use jq's @sh to generate shell-escaped (and thus eval-safe) output. Not sure if that was present as of this answer's 2014 original posting. :)

[HelpNeeder]

Thanks for pointing out my flaws. I do agree I'm not a professional bash programmer. Just needed to botch something really quick, and decided to share it.

[hmalphettes]

Many thanks for this. Here are some tweaks to remove some pipes and sanitize the keys: local -A associativeArray="$(echo "$json" | jq -r 'to_entries[] | "[" + (.key|@sh) + "]=" + (.value | @sh)'"

[Sid]

@hmalphettes Good additions. I would suggest replacing -r with j and adding a space at the end of the value string to get the same output as mine sans the wrapping parenthesis. I've updated my answer to reduce the complexity (replacing reduce with add) and include the @sh's.

[Thomas Praxl]

I like your solution very much. However, there is one flaw: using number values makes it think it has to recurse and then fail. It doesn't matter if the number is quoted in the json or not. Example: "alias": "XL" works, "geometry": "a1632" works, "sharpness": 100 doesn't work, "quality": "45" doesn't work. It fails with jq: error (at <stdin>:1): number (<number>) has no keys. The erroneous line is if jq -e . >/dev/null 2>&1 <<< "$PAIR_VALUE"; then

[HelpNeeder]

@Thomas Praxl, that's a great point. I didn't expect using datatype other than string, as I've used that snipped for simple log extraction. Hmm but seems like some sanitation could solve that problem without much effort.

[Cliff Armstrong]

Using double quotes (") to contain the jq filters is now considered bad practice. Use of single quotes (') is preferred to avoid issues with shell substitution of quoted strings. This way the line done < <(jq -r "to_entries|map(\"\(.key)=\(.value)\")|.[]" file) becomes done < <(jq -r 'to_entries|map("(.key)=(.value)")|.[]' file) which is more readable and less error prone.

[fedorqui]

@Cliff thank you! I just updated my answer with your suggestion

[Give_me_5_dolloars]

map("(.key)=(.value)") should be map("\(.key)=\(.value)")

[Tom N Tech]

-bash: (.key): syntax error: operand expected (error token is ".key)")

[Charles Duffy]

This is very comprehensive! That said, any particular reason for using legacy ksh instead of POSIX function syntax? See the entry in the last table in wiki.bash-hackers.org/scripting/obsolete discussing pros/cons.

[fedorqui]

@rjurney is jq installed in your system?

[Sid]

@CharlesDuffy Good catch. There is no reason besides having written it up fairly quickly. In my own code I use POSIX function syntax.

[Tom N Tech]

@fedorqui'SOstopharming' yes

[CrusherJoe]

Could you do this with a JSON array like "[ {"SALUTATION": "Hello world", "SOMETHING": "bla bla bla Mr. Freeman"},{"SALUTATION": "Hello world2", "SOMETHING": "bla bla bla Mr. Freeman2"} ]"? Then access the value like echo "${aa[1][SOMETHING]}."

[Ján Lalinský]

You reduced example does not work. bash: AA: "$(jq -r '. | to_entries | .[] | "[\"" + .key + "\"]=" + (.value | @sh)' <<< "$json")": must use subscript when assigning associative array

[Xavier Mol]

Theoretically yes, but practically no, because bash does not support multidimensional arrays.

[John Sick]

This answer is so good, and works like a charm. Just one more addition, here if you have a file e.g. ABC.json. Just change this line FILE=ABC.json aacontent=$(jq -r '. | to_entries | .[] | "[\"" + .key + "\"]=" + (.value | @sh)' <"$FILE")

[Skywarth]

I just liked your solution a lot so i made some additions to it stackoverflow.com/a/68518734/7204671

[Grisha Levit]

@JánLalinský the correct syntax should be local -A A="($(jq …))" rather than local -A A=("$(jq …)")

[pmg7670]

@Give_me_5_dolloars is correct .. looks weird, escaping only the left side of the brackets .. I suspect because of jq tokenized serial parsing !!

[Joshua Skrzypek]

@pmg7670 good guess, but it's actually just explicitly stated in the jq manual as the expected syntax: stedolan.github.io/jq/manual/#Stringinterpolation-(foo) –– A potential issue with it is that when you do use double quotes, you also need to escape the backslash. Thus you either need jq -r "to_entries|map(\"\\(.key)=\\(.value)\")|.[]" file (if you need to use double quotes, say to put a filter in a variable, as in @sid's answer below) or jq -r 'to_entries|map("\(.key)=\(.value)")|.[]' file.