Bash script -- store `curl` output in variable, then format against string in variable

Question

I was hoping to curl data into a shell script variable, then use cat or awk to operate on that variable as if it was a file.

Is this possible, or is there a workaround? Do I have to save the curl command into a file?

Do I have to run curl several times, for each bit of string formatting I want to do?

I am curling ipinfo.io, and want to output the matching lines for "city" and "region". I know how to use awk and sed to format the output as desired.

Try for example, curl ipinfo.io/"8.8.8.8" for the result for a Google DNS IP address.

I'd like to print just "Mountain View" and "California," preferably on the same line, with some common formatting, like "Mountain View, California".

Solution 1

In general there are more appropriate ways of parsing JSON objects, but since in this case the JSON object is very simple you may store curl's output in a variable (which is possible) and just use AWK:

var="$(curl ipinfo.io/"8.8.8.8" 2>/dev/null)"
<<<"$var" awk -F'"' '$2=="city"{printf("%s, ", $4)}$2=="region"{print $4}'

% var="$(curl ipinfo.io/"8.8.8.8" 2>/dev/null)"
% <<<"$var" awk -F'"' '$2=="city"{printf("%s, ", $4)}$2=="region"{print $4}'
Mountain View, California

However unless you want to use curl's output multiple times you may just use a pipe:

curl ipinfo.io/"8.8.8.8" 2>/dev/null | awk -F'"' '$2=="city"{printf("%s, ", $4)}$2=="region"{print $4}'

curl ipinfo.io/"8.8.8.8" 2>/dev/null | awk -F'"' '$2=="city"{printf("%s, ", $4)}$2=="region"{print $4}'
Mountain View, California

---

<<< is a form of input redirection called "here string"; it redirects the STDIN of a command from the terminal to a string.

What happens here is that $var is expanded between the double quotes; the STDIN of the AWK command is redirected from the terminal to the expanded string and AWK consequently reads the string as its input file.

Solution 2

I was hoping to curl data into a shell script variable, then use cat or awk to operate on that variable as if it were a file. Is this possible, or is there a workaround?

Sure you can. If the content is not too large, and if you want to run multiple commands to parse the content, then it's a good thing to cache it in memory rather than re-download every time.

To store the result of curl in a variable:

ipinfo=$(curl ipinfo.io/8.8.8.8)

To run commands on it:

<<< "$ipinfo" awk ...
<<< "$ipinfo" sed ...

The double-quotes no `"$ipinfo" are important to preserve all the whitespace characters.

A "workaround" to not saving the content in a variable is to figure out a way to process the content in a single pipeline, like @kos did.

Solution 3

Another easy and more readable way to do this, you can use python, which exists in all linux based systems, to parse the json for you

Simply, Import json and sys packages to load the json object, then print from the result object any property you need:

curl ipinfo.io/"8.8.8.8" 2>/dev/null | 
  python -c 'import  json,sys; 
             result=json.load(sys.stdin); 
             print(result["'city'"] + ", " + result["'region'"])';

I added the script in a more readable way however you should put in one line or add \ by the end of each line if you need to write it in multiple lines.

Comments

[heemayl]

Whats your input and desired output?

[kos]

You can, but there's no point in storing curl's output (unless you want to use it multiple times), you can directly feed it to cat or awk. As heemayl said please explain exactly what you want to do.

[cfye14]

i am curling ipinfo.io, and want to echo the matching lines for city and region. I know how to use awk and sed to format the output as desired. Try for example, curl ipinfo.io/"8.8.8.8" for the result for a google dns ip address

[kos]

What do you want to print exactly? The whole line as in the output or only "Mountain View" and "California"?

[cfye14]

just "Mountain View" and "California," preferably on the same line, with some common formatting, like you know, "Mountain View, California"

[cfye14]

can you explain the <<< part a bit more? edit: this I take it?

[kos]

@cfye14 I added an explanation for the <<< token: one thing, note that I asked all those informations to be extremely sure of the scope of your question (to avoid anyone giving potentially unuseful answers); I understand that the processing itself was probably not necessary. janos' answer below is just as much as valid.