How to split a string with quotes (like command arguments) in bash?

Question

I have a string like this:

"aString that may haveSpaces IN IT" bar foo "bamboo" "bam boo"

I want to be able to split it like this:

aString that may haveSpaces IN IT
bar
foo
bamboo  
bam boo

How do I do that? (preferrably using a one-liner)

Solution 1

When I saw David Postill's answer, I thought "there must be a simpler solution". After some experimenting I found the following works:-

string='"aString that may haveSpaces IN IT" bar foo "bamboo" "bam boo"'
echo $string
eval 'for word in '$string'; do echo $word; done'

This works because eval expands the line (removing the quotes and expanding string) before executing the resultant line (which is the in-line answer):

for word in "aString that may haveSpaces IN IT" bar foo "bamboo" "bam boo"; do echo $word; done

An alternative which expands to the same line is:

eval "for word in $string; do echo \$word; done"

Here string is expanded within the double-quotes, but the $ must be escaped so that word in not expanded before the line is executed (in the other form the use of single-quotes has the same effect). The results are:-

[~/]$ string='"aString that may haveSpaces IN IT" bar foo "bamboo" "bam boo"'
[~/]$ echo $string
"aString that may haveSpaces IN IT" bar foo "bamboo" "bam boo"
[~/]$ eval 'for word in '$string'; do echo $word; done'
aString that may haveSpaces IN IT
bar
foo
bamboo
bam boo
[~/]$ eval "for word in $string; do echo \$word; done"
aString that may haveSpaces IN IT
bar
foo
bamboo
bam boo

Solution 2

The simplest solution is using making an array of the quoted args which you could then loop over if you like or pass directly to a command.

eval "array=($string)"

for arg in "${array[@]}"; do echo "$arg"; done   

p.s. Please comment if you find a simpler way without eval.

Edit:

Building on @Hubbitus' answer we have a fully sanitized and properly quoted version. Note: this is overkill and will actually leave additional backslashes in double or single quoted sections preceding most punctuation but is invulnerable to attack.

declare -a "array=($( echo "$string" | sed 's/[][`~!@#$%^&*():;<>.,?/\|{}=+-]/\\&/g' ))"

I leave it to the interested reader to modify as they see fit http://ideone.com/FUTHhj

Solution 3

It looks that xargs can do it pretty well:

$ a='"aString that may haveSpaces IN IT" bar foo "bamboo" "bam boo"'
$ printf "%s" "$a" | xargs -n 1 printf "%s\n"
aString that may haveSpaces IN IT
bar
foo
bamboo
bam boo

Solution 4

How do I do that?

$ for l in "aString that may haveSpaces IN IT" bar foo "bamboo" "bam boo"; do echo $l; done
aString that may haveSpaces IN IT
bar
foo
bamboo
bam boo

---

What do I do if my string is in a bash variable?

The simple approach of using the bash string tokenizer will not work, as it splits on every space not just the ones outside quotes:

DavidPostill@Hal /f/test
$ cat ./test.sh
#! /bin/bash
string='"aString that may haveSpaces IN IT" bar foo "bamboo" "bam boo"'
for word in $string; do echo "$word"; done

DavidPostill@Hal /f/test
$ ./test.sh
"aString
that
may
haveSpaces
IN
IT"
bar
foo
"bamboo"
"bam
boo"

To get around this the following shell script (splitstring.sh) shows one approach:

#! /bin/bash 
string=$(cat <<'EOF'
"aString that may haveSpaces IN IT" bar foo "bamboo" "bam boo" 
EOF
)
echo Source String: "$string"
results=()
result=''
inside=''
for (( i=0 ; i<${#string} ; i++ )) ; do
    char=${string:i:1}
    if [[ $inside ]] ; then
        if [[ $char == \\ ]] ; then
            if [[ $inside=='"' && ${string:i+1:1} == '"' ]] ; then
                let i++
                char=$inside
            fi
        elif [[ $char == $inside ]] ; then
            inside=''
        fi
    else
        if [[ $char == ["'"'"'] ]] ; then
            inside=$char
        elif [[ $char == ' ' ]] ; then
            char=''
            results+=("$result")
            result=''
        fi
    fi
    result+=$char
done
if [[ $inside ]] ; then
    echo Error parsing "$result"
    exit 1
fi

echo "Output strings:"
for r in "${results[@]}" ; do
    echo "$r" | sed "s/\"//g"
done

Output:

$ ./splitstring.sh
Source String: "aString that may haveSpaces IN IT" bar foo "bamboo" "bam boo"
Output strings:
aString that may haveSpaces IN IT
bar
foo
bamboo
bam boo

Source: StackOverflow answer Split a string only by spaces that are outside quotes by choroba. Script has been tweaked to match the requirements of the question.

Solution 5

You may do it with declare instead of eval, for example:

Instead of:

string='"aString that may haveSpaces IN IT" bar foo "bamboo" "bam boo"'
echo "Initial string: $string"
eval 'for word in '$string'; do echo $word; done'

Do:

declare -a "array=($string)"
for item in "${array[@]}"; do echo "[$item]"; done

But please note, it is not much safer if input comes from user!

So, if you try it with say string like:

string='"aString that may haveSpaces IN IT" bar foo "bamboo" "bam boo" `hostname`'

You get hostname evaluated (there off course may be something like rm -rf /)!

Very-very simple attempt to guard it just replace chars like backtrick ` and $:

string='"aString that may haveSpaces IN IT" bar foo "bamboo" "bam boo" `hostname`'
declare -a "array=( $(echo $string | tr '`$<>' '????') )"
for item in "${array[@]}"; do echo "[$item]"; done

Now you got output like:

[aString that may haveSpaces IN IT]
[bar]
[foo]
[bamboo]
[bam boo]
[?hostname?]

More details about methods and pros and cons you may found in that good answer: https://stackoverflow.com/questions/17529220/why-should-eval-be-avoided-in-bash-and-what-should-i-use-instead/17529221#17529221

But there still leaved vector for attack. I very want have in bash method of string quote like in double quotes (") but without interpreting content.

View more solutions

Comments

[DavidPostill]

Stack Overflow duplicate: Split a string only by spaces that are outside quotes

[foxneSs]

@DavidPostill the questions are quite different actually.

[DavidPostill]

Not really, it's the same general problem.

[AFH]

@DavidPostill - This is a much simpler problem: all it needs is for l in "aString that may haveSpaces IN IT" bar foo "bamboo" "bam boo"; do echo $l; done

[DavidPostill]

@AFH lol. I just posted a much longer answer. The only difference in the output was that mine preserved the "s. I missed the fact that the OP doesn't need them in the output.

[DavidPostill]

@AFH You should post your comment as the answer.

[AFH]

@DavidPostill - It's more complicated if the string is in a variable. If the string is in $s, then for l in $s; do echo $l; done takes the quotes as literals and breaks at the spaces. I need to go out now, so feel free to work it out.

[barlop]

it's called tokenizing a string.. e.g. modern programming/scripting languages / libraries, have a string tokenizer facility. For bash stackoverflow.com/questions/5382712/…

[DavidPostill]

@barlop The tokening in the linked question splits on every space not just the ones outside quotes.

[Flamefire]

I have a testcase that breaks this: string="bash -c 'echo \$USER'" which "leaves" the backslash. You sometimes need this for e.g ssh