Error when running curl inside script (No such file or directory)

Question

I have the following .sh file I pieced together off of things I found on the internet.
The goal is to read the 2nd line 2nd item in a CSV file and use that to send a delete commmand to elasticsearch.

#!/bin/sh
OLDIFS=$IFS
IFS=,


function quit {
        echo "Quitting Script"
        IFS=$OLDIFS
    exit 1
}
function fileExists {
    if [ ! -f "$1" ]
        then
            echo "File $1 does not exists"
                quit
        fi
echo "File Name:  $1"
}
function work {
        linesToSkip=1
        {
            for ((i=$linesToSkip;i--;)) ;do
                read
            done
                #Read 2nd item of the 2nd line of CSV file to get PROGRAMURL
            read INDEX PROGRAMURL JUNK
                echo "$PROGRAMURL"
                QUERY="curl -XDELETE http://127.0.0.1:9200/cj/_query  -d '{ \"query\" : { \"match\" : { \"PROGRAMURL\" : "$PROGRAMURL" } } }'"
                $("$QUERY")
                #RESPONSE=`$QUERY`
                #echo $RESPONSE

        } < $1

}

fileExists $1
work $1
IFS=$OLDIFS

Everything works except the execution of the curl script. I've tried it with $(), backtics, exec, and I cannot get it to work.

The following is the error when I run bash -vx script.sh:

bash -vx ./deleteExisting.sh catalog.csv
#!/bin/sh
OLDIFS=$IFS
+ OLDIFS='  
'
IFS=,
+ IFS=,


function quit {
    echo "Quitting Script"
    IFS=$OLDIFS
    exit 1
}
function fileExists {
    if [ ! -f "$1" ]
    then
        echo "File $1 does not exists"
        quit
    fi  
echo "File Name:  $1"
}
function work {
    linesToSkip=1
    {
        for ((i=$linesToSkip;i--;)) ;do
            read
        done
        #Read 2nd item of the 2nd line of CSV file to get PROGRAMURL
        read INDEX PROGRAMURL JUNK
        echo "$PROGRAMURL"
        QUERY="curl -XDELETE http://127.0.0.1:9200/cj/_query  -d '{ \"query\" : { \"match\" : { \"PROGRAMURL\" : "$PROGRAMURL" } } }'"
        $("$QUERY")
        #RESPONSE=`$QUERY`
        #echo $RESPONSE

    } < $1  

}

fileExists $1
+ fileExists catalog.csv
+ '[' '!' -f catalog.csv ']'
+ echo 'File Name:  catalog.csv'
File Name:  catalog.csv
work $1
+ work catalog.csv
+ linesToSkip=1
+ (( i=1 ))
+ (( i-- ))
+ read
+ (( 1 ))
+ (( i-- ))
+ read INDEX PROGRAMURL JUNK
+ echo '"http://www.website.com"'
"http://www.website.com"
+ QUERY='curl -XDELETE http://127.0.0.1:9200/cj/_query  -d '\''{ "query" : { "match" : { "PROGRAMURL" : "http://www.website.com" } } }'\'''
"$QUERY")
"$QUERY"
++ 'curl -XDELETE http://127.0.0.1:9200/cj/_query  -d '\''{ "query" : { "match" : { "PROGRAMURL" : "http://www.website.com" } } }'\'''
./deleteExisting.sh: line 29: curl -XDELETE http://127.0.0.1:9200/cj/_query  -d '{ "query" : { "match" : { "PROGRAMURL" : "http://www.website.com" } } }': No such file or directory
IFS=$OLDIFS 
+ IFS='     
'

Example CSV file would be

INDEX, PROGRAMURL, other, info
"1", "http://website.com", "other info", "information"

Solution

Running $("$QUERY") actually tries to run in a sub-shell, a command called <the expanded value of $QUERY>, hence your '(bunch of stuff) No such file or directory' error.

You probably want something like:

CURLURL="http://127.0.0.1:9200/cj/_query"
CURLDATA='{ "query" : { "match" : { "PROGRAMURL" : "'$PROGRAMURL'" } } }'
RESPONSE=`curl -XDELETE "$CURLURL"  -d "$DATA"`

The trick here is how you nest the single and double quotes. Its a bit hard to explain concisely, but here goes:

• Shell variables do not expand inside an outer single quote.
• But you dont need to escape a double quote inside a single quote
• So how we get $PROGRAMURL to expand above is immediately concatenate it between a pair of closing-opening single quotes.
• To subsequently use the variable CURLDATA it has to get passed to the curl command inside double quotes
• A much simpler example:

VARIABLE1="Hello, world"
VARIABLE2='This is "$verbatim" stuff"'$VARIABLE1'" More stuff'

• This produces a value of $VARIABLE2 of This is "$verbatim" stuff"Hello, World" More stuff

Other things to note:

• The above will put the entire stdout of the curl command into the variable RESPONSE
• You will need to check $? afterwards to see if curl actually managed to talk to the host, etc. - it will be 0 if everything went OK
• You may wish to turn off the progress bar

Skipping lines:

• A better way to skip lines is to use:

tail -n +$(( $linesToSkip +1 ))

Comments

[goldie4]

Looks like that worked, thank you for the explanation.

[goldie4]

I updated the script with what you had said, and I'm having the same issue again, I think I'm doing it correctly.. Would you mind looking at it? stackoverflow.com/questions/22759076/… Thanks

[Pinak Mazumdar]

Excellent Stuff :) god bless you.