jq & bash: make JSON array from variable

Question

I'm using jq to form a JSON in bash from variable values.

Got how to make plain variables

$ VAR="one two three"
$ jq -n "{var:\"$VAR\"}"
{
  "var": "one two three"
}

But can't make arrays yet. I have

$ echo $ARR
one
two
three

and want to get something like

{
  "arr": ["one", "two", "three"]
}

I only manage to get garbled output like

$ jq -n "{arr: [\"$ARR\"]}"
{
  "arr": [
    "one\ntwo\nthree"
  ]
}

How to form JSON array in a correct way? Can jq ever do that?

EDIT: Question was asked when there was only jq 1.3. Now, in jq 1.4, it is possible to do straightly what I asked for, like @JeffMercado and @peak suggested, upvote for them. Won't undo acceptance of @jbr 's answer though.

Solution 1

jq is doing exactly what you tell it to do. jq is not a program for generating JSON, but a tool for querying it. What you are doing with the -n switch is just using it as a pretty printer. So if you want it to print an object with an array containing "one", "two", "three" then you have to generate it.

VAR="one two three"
VAR=$(echo $VAR | sed -e 's/\(\w*\)/,"\1"/g' | cut -d , -f 2-)
echo "{var: [$VAR]}"

Update

As Bryan and others mention below it is indeed possible to generate JSON with jq, and from version 1.4, it's even possible to do what the OP ask directly, see for example Jeff's answer.

Solution 2

In jq 1.3 and up you can use the --arg VARIABLE VALUE command-line option:

jq -n --arg v "$VAR" '{"foo": $v}'

I.e., --arg sets a variable to the given value so you can then use $varname in your jq program, and now you don't have to use shell variable interpolation into your jq program.

EDIT: From jq 1.5 and up, you can use --argjson to pass in an array directly, e.g.

jq -n --argjson v '[1,2,3]' '{"foo": $v}'

Solution 3

Once you have your variable loaded, you should use the split filter to split that string into an array.

$ jq -n --arg inarr "${ARR}" '{ arr: $inarr | split("\n") }'

Solution 4

The original posting in this thread mentions VAR="one two three". For such a variable, a very simple solution in jq 1.4 is as follows:

$ jq -n -c -M --arg s "$VAR" '{var: ($s|split(" "))}'
{"var":["one","two","three"]}

(This assumes a Linux or Mac shell or similar, and works with jq 1.4.)

If the separator character is a newline, then consider this typescript:

$ s=$(printf "a\nb\nc\n")
$ jq -n -c -M --arg var "$s" '{"var": ($var|split("\n"))}'
{"var":["a","b","c"]}

Notice that the JSON string for the newline character is "\n" (not "\\n").

A third case of interest is when VAR is a bash array. In this case, it may be acceptable to pass in the items using "${VAR[@]}", but in general, it must be remembered that the "--arg name value" option is intended only for passing in strings.

In jq>1.4, exported shell variables can be passed in using the "env" object, as documented in the online jq manual page.

Solution 5

This worked for me in jq 1.6:

$ jq -nc '$ARGS.positional' --args 1 2 3
["1","2","3"]

For this specific use-case, you could use an array and write:

$ VAR=(one two three)
$ jq -nc '{var: $ARGS.positional}' --args ${VAR[@]}
{"var":["one","two","three"]}

You could also use a string like this:

$ VAR="one two three"
$ jq -nc '{var: ($ARGS.positional[0] | split(" "))}' --args $VAR

View more solutions

Comments

[David Ongaro]

That works only with gnu sed. I suggest replacing the second line with VAR=$(printf '"%s"\n' $VAR|paste -sd, -).

[Bryan Larsen]

jq is a fabulous way of generating JSON, especially if you may have any sort of special characters (like ") in the values you want to write to the JSON

[peak]

As Bryan Larsen wrote, jq is an excellent choice for generating JSON, as illustrated in some of the responses below. Could someone remove the erroneous information in the post above, and/or down-vote it (as it doesn't not answer the question)?

[Andrey Regentov]

this string began to work for me when i wrote --arg ARR "$ARR" instead of --arg ARR $ARR.

[Faron]

@DavidOngaro -- +1 for your comment. That was important since that command wraps the whole string as a variable. I.e. Original command line - all of my files were broken-listed as filename AND extension individually as shown: { files:["filename".,"mp4","filename2".,"mp4"]}. The correct method (your suggestion) showed as { files: ["filename.mp4","filename2.mp4"]}. Thank you.

[gazarsgo]

Did you try it? Just use '[$v]' instead.

[ghoti]

@AndreyRegentov - probably not. If jq is reading the environment, then a variable must be EXPORTed (in bash) to be visible, and requests to export arrays are silently ignored. The way for any program called by bash to "receive" the contents of an array would be to express it as a string, with field separators. You might try jq -n --argjson v "[$(printf '"%s",' "${arr[@]}")0]" '{"foo": $v}', which only leaves a little garbage in the array, and fails if the array data contains quotes.

[jerney]

To possibly save someone else time in the future: The -c option compacts the output and the -M option makes the output monochrome. They do not affect the core functionality

[HappyFace]

--compact-output is also useful.