String to integer in Shell

Question

I'm currently working on a project called 'dmCLI' which means 'download manager command line interface'. I'm using curl to multi-part download a file. And I'm having trouble with my code. I can't convert string to int.

Here's my full code. I also uploaded my code into Github Repo. Here it is: dmCLI GitHub Repository

#!/bin/bash
RED='\033[0;31m'
NC='\033[0m'

help() {
    echo "dmcli [ FILENAME:path ] [ URL:uri ]"
}

echo -e "author: ${RED}@atahabaki${NC}"
echo -e "     __      _______   ____"
echo -e " ___/ /_ _  / ___/ /  /  _/"
echo -e "/ _  /  ' \/ /__/ /___/ /  "
echo -e "\_,_/_/_/_/\___/____/___/  "
echo -e "                           "
echo -e "${RED}Downloading${NC} has never been ${RED}easier${NC}.\n"

if [ $# == 2 ]
then
    filename=$1;
    url=$2
    headers=`curl -I ${url} > headers`
    contentlength=`cat headers | grep -E '[Cc]ontent-[Ll]ength:' | sed 's/[Cc]ontent-[Ll]ength:[ ]*//g'`
    acceptranges=`cat headers | grep -E '[Aa]ccept-[Rr]anges:' | sed 's/[Aa]ccept-[Rr]anges:[ ]*//g'`
    echo -e '\n'
    if [ "$acceptranges" = "bytes" ]
    then 
        echo File does not allow multi-part download.   
    else
        echo File allows multi-part download.
    fi
    echo "$(($contentlength + 9))"
    # if [acceptranges == 'bytes']
    # then
    # divisionresult = $((contentlength/9))
    # use for to create ranges...
else 
    help
fi

# First get Content-Length via regex or request,
# Then Create Sequences,
# After Start Downloading,
# Finally, re-assemble to 1 file.

I want to divide contentlength's value by 9. I tried this:

echo "$(($contentlength/9))"

It's getting below error:

/9")syntax error: invalid arithmetic operator (error token is "

I'm using localhost written in node.js. I added head responses. It returns Content-Length of the requested file, and the above dmCLI.sh gets correctly the Content-Length value.

./dmcli.sh home.html http://127.0.0.1:404/

A HEAD request to http://127.0.0.1:404/ returns: Content-Length: 283, Accept-Range: bytes

The dmCLI works for getting the value, but when I want to access its value, it won't work.

Simple actions work like:

echo "$contentlength"

But I can't access it by using this:

echo "$contentlength bytes"

and here:

echo "$(($contentlength + 9))"

returns 9 but I'm expecting 292. Where's the problem; why is it not working?

Solution 1

The regex below will extract the number of bytes, only the number:

contentlength=$(
  LC_ALL=C sed '
    /^[Cc]ontent-[Ll]ength:[[:blank:]]*0*\([0-9]\{1,\}\).*$/!d
    s//\1/; q' headers
)

After the above change, the contentlength variable will be only made of decimal digits (with leading 0s removes so the shell doesn't consider the number as octal), thus the 2 lines below will display the same result:

echo "$(($contentlength/9))"

echo "$((contentlength/9))"

Solution 2

headers=`curl -I ${url} > headers`
contentlength=`cat headers | grep -E '[Cc]ontent-[Ll]ength:' | sed 's/[Cc]ontent-[Ll]ength:[ ]*//g'`
echo "$(($contentlength/9))"

It's getting below error:

/9")syntax error: invalid arithmetic operator (error token is "

HTTP headers always end with CR/LF, not just LF; your `cat headers | ...` command expansion will remove the ending LF, but leave the CR alone, which will cause that strange error.

% var=`printf "%s\r\n" 333`
% echo var={$var}
}ar={333
% echo "$((var / 3))"
")syntax error: invalid arithmetic operator (error token is "

Comments

[Weijun Zhou]

If echo "$contentlength" works, there is no reason echo "$contentlength bytes" does not work as well.

[Stéphane Chazelas]

Could be the value of the variable ends in a CR character (line delimiters in HTTP headers are CRLF), 12345\r bytes would show up as " bytes" in a terminal as CR moves the cursor to the beginning of the line.

[Weijun Zhou]

@StéphaneChazelas You're right. CR does not have this behavior when redirected. One can redirect the output and examine the result.

[A. Taha Baki]

echo " $contentlength bytes" it returns this: ``` bytes```

[Stéphane Chazelas]

The way it's written (not proper header parsing, no sanitisation of the content length), this script has an arbitrary command injection vulnerability. Do not use a shell for this kind of thing. perl/python have HTTP or libcurl APIs.

[Weijun Zhou]

If @StéphaneChazelas is correct, it does not return "bytes", it just show up as "bytes", pipe the output through cat -A or less to examine.

[A. Taha Baki]

"contentlength=cat headers | grep -E '[Cc]ontent-[Ll]ength:' | sed 's/[Cc]ontent-[Ll]ength:[ ]*//g' | cat -A" retuns "283^M$ bytes"

[A. Taha Baki]

It worked but what's the difference between echo $((contentlength/9)) and echo $(($contentlength))?

[Weijun Zhou]

I won't call $(($contentlength/9)) incorrect, just that the $ is unnecessary, but there are cases where this $ is needed.

[Jay jargot]

There will be ARITHMETIC EVALUATION with this syntax, you can read this in the man bash: Within an expression, shell variables may also be referenced by name without using the parameter expansion syntax.. In other words: do not use $ for variable with this syntax.

[Weijun Zhou]

But what if parameter expansion is needed? $ in $((${b/5/7}+1)) is needed.

[Jay jargot]

The answer had been edited.

[Stéphane Chazelas]

@Jayjargot, there will be arithmetic evaluation with both syntax in ksh, bash and zsh (hence the command injection vulnerabilities when the input is not sanitized), one difference is when the value is like 9+9. Then contentlength/9 will give 2 and $contentlength/9 will give 1.

[Jay jargot]

@StéphaneChazelas: ha ok, I never used the $(( )) with variables containing something else than a number: string 9+9 for example.

[Weijun Zhou]

@StéphaneChazelas $contentlength/9 gives 10 in bash.

[Jay jargot]

@StéphaneChazelas: true! answer Edited again

[Jay jargot]

@WeijunZhou: you were right about the parameter expansion, the answer had been edited again.

[Stéphane Chazelas]

@WeijunZhou, yes sorry my bad 9+9/9 is 10 not 1, (9+9)/9 is 2.

[Jay jargot]

@StéphaneChazelas: thx for the final version