Linux Shell - How to remove escape characters generated by JQ when json is read with sed?

Question

I am trying to read a json file and use its output in jq to build another json, combining so said json file and other values, and pass it to CURL

I read the file with policies=$( sed 's/\\//g' policies.json)

I build the new json with the following jq command

BODY=$( jq -n \
            --arg cid "$chaincodeId" \
            --arg cv "$chaincodeV" \
            --arg ct "$chaincodeT" \
            --arg ar "$chaincodeArgs" \
            --arg pol "$policies" \
            '{chaincode_id: $cid, chaincode_version: $cv, chaincode_type: $ct, endorsement_policy: $pol}' )

It works fine for the first 4 properties. However, the endorsement_policy property contains backslashes, and so the server fails to read property sent via curl.

The output of body is the following

{ "chaincode_id": "IdentityManager", "chaincode_version": "testcc2", "chaincode_type": "node", "endorsement_policy": "{\n \"identities\": [\n {\n \"role\": {\n \"name\": \"member\",\n \"mspId\": \"org1\"\n }\n },\n {\n \"role\": {\n \"name\": \"member\",\n \"mspId\": \"org2\"\n }\n },\n {\n \"role\": {\n \"name\": \"member\",\n \"mspId\": \"org3\"\n }\n }\n ],\n \"policy\": {\n \"1-of\": [\n {\n \"signed-by\": 0\n },\n {\n \"signed-by\": 1\n },\n {\n \"signed-by\": 2\n }\n ]\n }\n}" }

I don't understand how I can force jq to not generate backslashes, as the documentation explicitely says it will consider the variable as a string. Could someone provide me a hint for the solution?

Solution

When using --arg to pass data into a jq variable, the data in the variable will always be treated as a string. When a JSON fragment is passed in this way, jq will naturally escape the bits of that string that needs to be escaped so that it's a valid JSON-encoded string.

If you are passing a JSON fragment that you want to be used as JSON, not as a text string, then use --argjson instead of --arg.

In short, it looks like you should be using

BODY=$( jq -n \
            --arg cid "$chaincodeId" \
            --arg cv "$chaincodeV" \
            --arg ct "$chaincodeT" \
            --arg ar "$chaincodeArgs" \
            --argjson pol "$policies" \
            '{chaincode_id: $cid, chaincode_version: $cv, chaincode_type: $ct, endorsement_policy: $pol}' )

Also note that if your original policies.json file is a properly encoded JSON file, you should not be deleting backslashes from it.

To have the policies.json document inserted directly, you could possibly use

BODY=$( jq -n \
            --arg cid "$chaincodeId" \
            --arg cv "$chaincodeV" \
            --arg ct "$chaincodeT" \
            --arg ar "$chaincodeArgs" \
            --slurpfile pol policies.json \
            '{chaincode_id: $cid, chaincode_version: $cv, chaincode_type: $ct, endorsement_policy: $pol[] }' )

(note the $pol[]), but I haven't used this very often so I'm not 100% sure under what circumstances it breaks.

See also the "Invoking jq" section of the jq manual.

Comments

[Adrián]

Thank you for the argjson I didn't know this and it helped me today!