Using curl POST with variables defined in bash script functions

Question

When I echo I get this, which runs when I enter it into the terminal

curl -i \
-H "Accept: application/json" \
-H "Content-Type:application/json" \
-X POST --data '{"account":{"email":"[email protected]","screenName":"akdgdtk","type":"NIKE","passwordSettings":{"password":"Starwars1","passwordConfirm":"Starwars1"}},"firstName":"Test","lastName":"User","middleName":"ObiWan","locale":"en_US","registrationSiteId":"520","receiveEmail":"false","dateOfBirth":"1984-12-25","mobileNumber":"9175555555","gender":"male","fuelActivationDate":"2010-10-22","postalCode":"10022","country":"US","city":"Beverton","state":"OR","bio":"This is a test user","jpFirstNameKana":"unsure","jpLastNameKana":"ofthis","height":"80","weight":"175","distanceUnit":"MILES","weightUnit":"POUNDS","heightUnit":"FT/INCHES"}' https://xxx:[email protected]/xxxxx/xxxx/xxxx

But when run in the bash script file, I get this error

curl: (6) Could not resolve host: application; nodename nor servname provided, or not known
curl: (6) Could not resolve host: is; nodename nor servname provided, or not known
curl: (6) Could not resolve host: a; nodename nor servname provided, or not known
curl: (6) Could not resolve host: test; nodename nor servname provided, or not known
curl: (3) [globbing] unmatched close brace/bracket at pos 158

this is the code in the file

curl -i \
-H '"'Accept: application/json'"' \
-H '"'Content-Type:application/json'"' \
-X POST --data "'"'{"account":{"email":"'$email'","screenName":"'$screenName'","type":"'$theType'","passwordSettings":{"password":"'$password'","passwordConfirm":"'$password'"}},"firstName":"'$firstName'","lastName":"'$lastName'","middleName":"'$middleName'","locale":"'$locale'","registrationSiteId":"'$registrationSiteId'","receiveEmail":"'$receiveEmail'","dateOfBirth":"'$dob'","mobileNumber":"'$mobileNumber'","gender":"'$gender'","fuelActivationDate":"'$fuelActivationDate'","postalCode":"'$postalCode'","country":"'$country'","city":"'$city'","state":"'$state'","bio":"'$bio'","jpFirstNameKana":"'$jpFirstNameKana'","jpLastNameKana":"'$jpLastNameKana'","height":"'$height'","weight":"'$weight'","distanceUnit":"MILES","weightUnit":"POUNDS","heightUnit":"FT/INCHES"}'"'" "https://xxx:[email protected]/xxxxx/xxxx/xxxx"

I assume there's an issue with my quotation marks, but I've played with them a lot and I've gotten similar errors. All the variables are defined with different functions in the actual script

Solution 1

You don't need to pass the quotes enclosing the custom headers to curl. Also, your variables in the middle of the data argument should be quoted.

First, write a function that generates the post data of your script. This saves you from all sort of headaches concerning shell quoting and makes it easier to read an maintain the script than feeding the post data on curl's invocation line as in your attempt:

generate_post_data()
{
  cat <<EOF
{
  "account": {
    "email": "$email",
    "screenName": "$screenName",
    "type": "$theType",
    "passwordSettings": {
      "password": "$password",
      "passwordConfirm": "$password"
    }
  },
  "firstName": "$firstName",
  "lastName": "$lastName",
  "middleName": "$middleName",
  "locale": "$locale",
  "registrationSiteId": "$registrationSiteId",
  "receiveEmail": "$receiveEmail",
  "dateOfBirth": "$dob",
  "mobileNumber": "$mobileNumber",
  "gender": "$gender",
  "fuelActivationDate": "$fuelActivationDate",
  "postalCode": "$postalCode",
  "country": "$country",
  "city": "$city",
  "state": "$state",
  "bio": "$bio",
  "jpFirstNameKana": "$jpFirstNameKana",
  "jpLastNameKana": "$jpLastNameKana",
  "height": "$height",
  "weight": "$weight",
  "distanceUnit": "MILES",
  "weightUnit": "POUNDS",
  "heightUnit": "FT/INCHES"
}
EOF
}

It is then easy to use that function in the invocation of curl:

curl -i \
-H "Accept: application/json" \
-H "Content-Type:application/json" \
-X POST --data "$(generate_post_data)" "https://xxx:[email protected]/xxxxx/xxxx/xxxx"

This said, here are a few clarifications about shell quoting rules:

The double quotes in the -H arguments (as in -H "foo bar") tell bash to keep what's inside as a single argument (even if it contains spaces).

The single quotes in the --data argument (as in --data 'foo bar') do the same, except they pass all text verbatim (including double quote characters and the dollar sign).

To insert a variable in the middle of a single quoted text, you have to end the single quote, then concatenate with the double quoted variable, and re-open the single quote to continue the text: 'foo bar'"$variable"'more foo'.

Solution 2

Solution tested with https://httpbin.org/ and inline bash script
1. For variables without spaces in it i.e. 1:
Simply add ' before and after $variable when replacing desired string

for i in {1..3}; do \
  curl -X POST -H "Content-Type: application/json" -d \
    '{"number":"'$i'"}' "https://httpbin.org/post"; \
done

2. For input with spaces:
Wrap variable with additional " i.e. "el a":

declare -a arr=("el a" "el b" "el c"); for i in "${arr[@]}"; do \
  curl -X POST -H "Content-Type: application/json" -d \
    '{"elem":"'"$i"'"}' "https://httpbin.org/post"; \
done

Wow works :)

Solution 3

Curl can post binary data from a file so I have been using process substitution and taking advantage of file descriptors whenever I need to post something nasty with curl and still want access to the vars in the current shell. Something like:

curl "http://localhost:8080" \
-H "Accept: application/json" \
-H "Content-Type:application/json" \
--data @<(cat <<EOF
{
  "me": "$USER",
  "something": $(date +%s)
  }
EOF
)

This winds up looking like --data @/dev/fd/<some number> which just gets processed like a normal file. Anyway if you wanna see it work locally just run nc -l 8080 first and in a different shell fire off the above command. You will see something like:

POST / HTTP/1.1
Host: localhost:8080
User-Agent: curl/7.43.0
Accept: application/json
Content-Type:application/json
Content-Length: 43

{  "me": "username",  "something": 1465057519  }

As you can see you can call subshells and whatnot as well as reference vars in the heredoc. Happy hacking hope this helps with the '"'"'""""'''""''.

Solution 4

We can assign a variable for curl using single quote ' and wrap some other variables in double-single-double quote "'" for substitution inside curl-variable. Then easily we can use that curl-variable which here is MERGE.

Example:

# other variables ... 
REF_NAME="new-branch";

# variable for curl using single quote => ' not double "
MERGE='{
    "repository": "tmp",
    "command": "git",
    "args": [
        "pull",
        "origin",
        "'"$REF_NAME"'"
    ],
    "options": {
        "cwd": "/home/git/tmp"
    }
}';

notice this line:

    "'"$REF_NAME"'"

so we can use this bash variable $MERGE and calling curl as usual:

curl -s -X POST localhost:1365/M -H 'Content-Type: application/json' --data "$MERGE" 

Solution 5

A few years late but this might help someone if you are using eval or backtick substitution:

postDataJson="{\"guid\":\"$guid\",\"auth_token\":\"$token\"}"

Using sed to strip quotes from beginning and end of response

$(curl --silent -H "Content-Type: application/json" https://${target_host}/runs/get-work -d ${postDataJson} | sed -e 's/^"//' -e 's/"$//')

View more solutions

Comments

[Usman]

"'"$<variable name>"'" solved my problem where I needed quotes to be not omitted. Thanks.

[twistedstream]

This solution works but I think you can emit the extra double quotes surrounding the variable. So instead of this: --data '{"account": {"email": "'"$email"'"} }' you can do this: --data '{"account": {"email": "'$email'"} }'

[0rkan]

The other answer didn't work for me as I was trying to invoke it in an alert from Zabbix. This one solves it perfectly and is more clean.

[Hanynowsky]

But what if you put the code in a bash function : myFunction () { .... } ?

[Sir Athos]

@twistedstream for properly formatted e-mail addresses, yes. If your variable contains anything naughty though (e.g. spaces), omitting the extra quotes will break your script.

[dKab]

how to do the same on Windows?

[Sir Athos]

@dKab if you are on Windows 10, use bash and curl from the Linux Subsystem; otherwise, find one of the Windows ports for curl.

[Vasyl Boroviak]

Does not work for when when $i contains spaces. :(

[pbaranski]

Can you post an example?

[Vasyl Boroviak]

Sure. i="a b" instead of for-loop

[Vasyl Boroviak]

I found that the accepted and the second voted answer does not work in /bin/sh. However, this answer did the trick. And it's much simpler than the other answers. Thank you so much! I've edited your answer with some nicer line wrapping formatting. Otherwise, it's hard to spot the brilliance. Cheers mate

[user3053230]

Am I correct that the reason the quoting works is because the single-quotes are paired like so: '{"number":"' # the first pair of single-quotes, this has the opening double-quote for json syntax. $i # bash does variable substitution. '"}' # the next pair of single-quotes, this has the closing double-quote for json syntax.

[Vader B]

it's worth to note that this recipe works only if the script is copied verbatim (i.e. no reformatting EOF, braces etc.)

[Scrambo]

First off, thank you @SirAthos. This saved my butt. Secondly for everyone else that finds this, make sure you include the Content-Type:application/json It's easy to forget to put in, but it's necessary.

[Klaas]

Did not work when there was a space after the second EOF: EOF . After removing it everything is fine.

[jamis0n]

I needed to urlencode, so this worked for me with an endpoint expecting a payload: --data-urlencode payload="$(generate_post_data)"

[dbreaux]

I apologize for my ignorance, but where am I defining that generate_post_data() function?

[Sir Athos]

@dbreaux That depends where you run the curl command. If the command is in a script, you simply define the function anywhere above it in that same script. If you are running curl directly from the command line, you have several options, one of which is to type up the function in a new file and then at the command line run source my_new_file to define the function in your current environment. After that you can run the curl command as indicated.

[slashdottir]

Please explain the cat EOF stuff

[sudhir tataraju]

Thanks a lot @pbaranski you saved lot of my time

[Sir Athos]

@slashdottir That's a bash feature called Here Documents. You can read about it in more detail at this link - in particular, check out Example 19-5. There is also already a full question about it here on SO.

[AATHITH RAJENDRAN]

3. for input with multi var which has spaces:'{"text": "1.'"$var1"'\n2.'"$var2"'"}'

[tripleee]

Of course that works, but then you have a pesky temporary file to clean up afterwards. This is only really useful on Windows, where you can't reliably interpolate variables into strings.

[harshavmb]

The most elegant answer I would say in terms of simplicity, readability..

[kisna]

For some reason, shell script was escaping quotes, the right combination of quotes for the placeholders did it for me. some_body='{"query":"/v1/'$some_id'","body":""}' curl --data "$some_body" 'https://some.url'

[Oussama Boumaad]

simply worked for me. Thanks a million