Add new element to existing JSON array with jq

Question

I want to append an element to an array in a JSON file using the jq``add command, but it's not working.

report-2017-01-07.json file:

{  
   "report": "1.0",
   "data": {  
      "date": "2010-01-07",
      "messages": [  
         {  
            "date": "2010-01-07T19:58:42.949Z",
            "xml": "xml_samplesheet_2017_01_07_run_09.xml",
            "status": "OK",
            "message": "metadata loaded into iRODS successfully"
         },
         {  
            "date": "2010-01-07T20:22:46.949Z",
            "xml": "xml_samplesheet_2017_01_07_run_09.xml",
            "status": "NOK",
            "message": "metadata duplicated into iRODS"
         },
         {  
            "date": "2010-01-07T22:11:55.949Z",
            "xml": "xml_samplesheet_2017_01_07_run_09.xml",
            "status": "NOK",
            "message": "metadata was not validated by XSD schema"
         }
      ]
   }
}

I am using this command:

$ cat report-2017-01-07.json 
| jq -s '.data.messages {"date": "2010-01-07T19:55:99.999Z", "xml": "xml_samplesheet_2017_01_07_run_09.xml", "status": "OKKK", "message": "metadata loaded into iRODS successfullyyyyy"}'
jq: error: syntax error, unexpected '{', expecting $end (Unix shell quoting issues?) at <top-level>, line 1:
.data.messages {"date": "2010-01-07T19:55:99.999Z", "xml": "xml_samplesheet_2017_01_07_run_09.xml", "status": "OKKK", "message": "metadata loaded into iRODS successfullyyyyy"}               
jq: 1 compile error

Here's how I want the output to look:

{
    "report": "1.0",
    "data": {
        "date": "2010-01-07",
        "messages": [{
            "date": "2010-01-07T19:58:42.949Z",
            "xml": "xml_samplesheet_2017_01_07_run_09.xml",
            "status": "OK",
            "message": "metadata loaded into iRODS successfully"
        }, {
            "date": "2010-01-07T20:22:46.949Z",
            "xml": "xml_samplesheet_2017_01_07_run_09.xml",
            "status": "NOK",
            "message": "metadata duplicated into iRODS"
        }, {
            "date": "2010-01-07T22:11:55.949Z",
            "xml": "xml_samplesheet_2017_01_07_run_09.xml",
            "status": "NOK",
            "message": "metadata was not validated by XSD schema"
        }, {
            "date": "2010-01-07T19:55:99.999Z",
            "xml": "xml_samplesheet_2017_01_07_run_09.xml",
            "status": "OKKKKKKK",
            "message": "metadata loaded into iRODS successfullyyyyy"
        }]
    }
}

Solution 1

The |= .+ part in the filter adds a new element to the existing array. You can use jq with filter like:

jq '.data.messages[3] |= . + {"date": "2010-01-07T19:55:99.999Z", "xml": "xml_samplesheet_2017_01_07_run_09.xml", "status": "OKKK", "message": "metadata loaded into iRODS successfullyyyyy"}' inputJson

To avoid using the hardcoded length value 3 and dynamically add a new element, use . | length which returns the length, which can be used as the next array index, i.e.,

jq '.data.messages[.data.messages| length] |= . + {"date": "2010-01-07T19:55:99.999Z", "xml": "xml_samplesheet_2017_01_07_run_09.xml", "status": "OKKK", "message": "metadata loaded into iRODS successfullyyyyy"}' inputJson

(or) as per peak's suggestion in the comments, using the += operator alone

jq '.data.messages += [{"date": "2010-01-07T19:55:99.999Z", "xml": "xml_samplesheet_2017_01_07_run_09.xml", "status": "OKKK", "message": "metadata loaded into iRODS successfullyyyyy"}]'

which produces the output you need:

{
  "report": "1.0",
  "data": {
    "date": "2010-01-07",
    "messages": [
      {
        "date": "2010-01-07T19:58:42.949Z",
        "xml": "xml_samplesheet_2017_01_07_run_09.xml",
        "status": "OK",
        "message": "metadata loaded into iRODS successfully"
      },
      {
        "date": "2010-01-07T20:22:46.949Z",
        "xml": "xml_samplesheet_2017_01_07_run_09.xml",
        "status": "NOK",
        "message": "metadata duplicated into iRODS"
      },
      {
        "date": "2010-01-07T22:11:55.949Z",
        "xml": "xml_samplesheet_2017_01_07_run_09.xml",
        "status": "NOK",
        "message": "metadata was not validated by XSD schema"
      },
      {
        "date": "2010-01-07T19:55:99.999Z",
        "xml": "xml_samplesheet_2017_01_07_run_09.xml",
        "status": "OKKK",
        "message": "metadata loaded into iRODS successfullyyyyy"
      }
    ]
  }
}

Use jq-play to dry-run your jq-filter and optimize any way you want.

Solution 2

Rather than using |=, consider using +=:

.data.messages += [{"date": "2010-01-07T19:55:99.999Z",
   "xml": "xml_samplesheet_2017_01_07_run_09.xml",
   "status": "OKKK", "message": "metadata loaded into iRODS successfullyyyyy"}]

Prepend

On the other hand, if (as @NicHuang asked) you want to add the JSON object to the beginning of the array, you could use the pattern:

 .data.messages |= [ _ ] + .

Solution 3

Summary: ". +" is your saviour

Details:

For adding an entry to a list: You can append [list1] + [list2] (and not [list] + data)

$ echo '[ "data1" ]' | jq '. + [ "data2" ]'
[
  "data1",
  "data2"
]

$ echo '[ {"key1": "value1"} ]' | jq '. + [{"key2": "value2"}]'
[
  {
    "key1": "value1"
  },
  {
    "key2": "value2"
  }
]

For adding a key/value to a dictionary:

$ echo '{"key1": "value1"}' | jq '. + {"key2": "value2"}'
{
  "key1": "value1",
  "key2": "value2"
}

References:

https://gist.github.com/joar/776b7d176196592ed5d8

Comments

[peak]

Since the goal is just to append an element, it would be better to use += but it is worth noting that .data.messages[3] |= . + X can here be simplified to .data.messages[3] = X as the . on the RHS is effectively just null.

[script'n'code]

(windows JQ user here): I wonder if it is possible to write this new data to the same (input) file, instead of creating a temporary file which needs to be renamed afterwords..

[peak]

@script'n'code - Using sponge is probably still the best option if you have it or can install it (it's part of moreutils).

[Nic Huang]

@peak how can I insert that object to the beginning of the array, couldn’t figure out . Thanks

[Harry Ma]

@peak how can I specify the order when inserting the element to the array ?

[Vitalii Zinchenko]

Prepend example: echo '{"arr": [ "data1" ]}' | jq '.arr |= ["data2"] + .' => {"arr": ["data2", "data1"]}