Expansion of a shell variable and effect of glob and split on it

Question

This post actually contains two separate questions but I think grouping them together will give some more context. I have gone through this question on quotes around variables but I do not quite understand what variable expansion mean in the first place. So my first question is :

1. What is variable expansion in unix/linux speak?

A second part of my question relates to the following terms :

1. glob
2. split

What do the above mean and how do they affect variable expansion? The answer to the original question mentions the following:

Think of the absence of quotes (in list contexts) as the split+glob operator.

As if echo $test was echo glob(split("$test")).

I couldn't find any answers that directly addresses the concept of globbing and splitting but rather uses those terms directly in answering other questions like this recent one.

Solution 1

Variable expansion (the standard term is parameter expansion, and it's also sometimes called variable substitution) basically means replacing the variable by its value. More precisely, it means replacing the $VARIABLE construct (or ${VARIABLE} or ${VARIABLE#TEXT} or other constructs) by some other text which is built from the value of the variable. This other text is the expansion of the variable.

The expansion process goes as follows. (I only discuss the common case, some shell settings and extensions modify the behavior.)

1. Take the value of the variable, which is a string. If the variable is not defined, use the empty string.
2. If the construct includes a transformation, apply it. For example, if the construct is ${VARIABLE#TEXT}, and the value of the variable begins with TEXT, remove TEXT from the beginning of the value.
3. If the context calls for a single word (for example within double quotes, or in the right-hand side of an assignment, or inside a here document), stop here. Otherwise continue with the next steps.
4. Split the value into separate words at each sequence of whitespace. (The variable IFS can be changed to split at characters other than whitespace.) The result is thus no longer a string, but a list of strings. This list can be empty if the value contained only whitespace.
5. Treat each element of the list as a file name wildcard pattern, i.e. a glob. If the pattern matches some files, it is replaces by the list of matching file names, otherwise it is left alone.

For example, suppose that the variable foo contains a* b* c* and the current directory contains the files bar, baz and paz. Then ${foo#??} is expanded as follows:

1. The value of the variable is the 8-character string a* b* c*.
2. #?? means strip off the first two characters, resulting in the 6-character string  b* c* (with an initial space).
3. If the expansion is in a list context (i.e. not in double quotes or other similar context), continue.
4. Split the string into whitespace-delimited words, resulting in a list of two-strings: b* and c*.
5. The string b*, interpreted as a pattern, matches two files: bar and baz. The string c* matches no file so it is left alone. The result is a list of three strings: bar, baz, c*.

For example echo ${foo#??} prints bar baz c* (the command echo joins its arguments with a space in between).

For more details, see:

• Parameter expansion in the POSIX standard, followed by field splitting and pathname expansion
• Shell parameter expansion in the bash manual, followed by word splitting and filename expansion
• $VAR vs ${VAR} and to quote or not to quote
• When is double-quoting necessary?

Solution 2

glob/split

I'll take the glob/split first. @Stephane's answer you linked to is using those terms in a general sense. They are not actual commands or anything like that, just pseudo operations.

The split("$test") would split the contents of "$test" up into an "array" of elements.

The glob(...) would then take care to expand any of these elements that contain shell globbing characters such as * or ranges [1-2].

Example

Say our string $test is as follows.

$ test="afile[1-2] afile[3-5]"

Also let's say we have a directory with some files in it.

$ ls -1
afile1
afile2
afile3
afile4
afile5

Now if we attempt to echo it without quotes you should notice that our string got split up on spaces, and then any globbing characters got expanded.

$ echo $test
afile1 afile2 afile3 afile4 afile5

However if we were to quote the variable when we passed it as an argument to echo we'd get the original literal string.

$ echo "$test"
afile[1-2] afile[3-5]

variable expansion

The term variable expansion is meant to cover the basic operation that the shell is performing as part of it's basic operations. The shell is responsible for parsing input and then executing this input once it's deemed syntactically correct.

In our previous example. When the variable $test was presented to echo unquoted we were telling the shell to go ahead and split those arguments up and then glob them.

When it was quoted, we were essentially disabling that feature with what ever variable(s) we wrapped with double quotes.

Example

Here are some additional examples of globbing and splitting.

glob/splitting happens automatically

$ echo file{1..3}
file1 file2 file3

$ echo file{1..3} dir{a..b}
file1 file2 file3 dira dirb

$ echo dir{z..w} file{A..D}
dirz diry dirx dirw fileA fileB fileC fileD

$ echo dir{z..w} file{A..B} fileC
dirz diry dirx dirw fileA fileB fileC

glob/splitting disabled via double quoting

$ echo "dir{z..w} file{A..B} fileC"
dir{z..w} file{A..B} fileC

$ echo "dir{z..w} file{A..B}"
dir{z..w} file{A..B}

Comments

[Stéphane Chazelas]

See also Security implications of forgetting to quote a variable in bash/POSIX shells

[Stéphane Chazelas]

Note that it's called p̲a̲r̲a̲m̲e̲t̲e̲r̲ expansion because it applies to variables ($var) and other types of parameters like $1, $#, $?, $-...

[gokhan acar]

I didn't know ranges also worked in reverse order.