What characters need to be escaped in files without quotes?

Question

I have browser-based shell/terminal that executes bash commands and I'm escaping spaces but it turns out that parenthesis also need to be escaped. What other characters need to be escaped for file names that are not in quotes?

Solution

The simple solution is to put a single quote (') at the beginning and another single quote at the end, and to replace every ' character inside the file name by the 4-character sequence '\''. All characters lose their special meaning inside a single-quoted string, except ' itself which marks the end of the string. The sequence '\'' ends the single-quoted literal, immediately follows by a quoted single quote, and opens a new single-quoted literal. Thus the file name

This file's name has some weird characters!
Will you manage to escape them?

can be quoted as follows:

somecommand 'This file'\''s name has some weird characters!
Will you manage to escape them?'

Double quotes have more complex escaping rules and don't allow you to include an exclamation mark ! if history substitution is activated, so I won't consider them further.

An alternative approach is to protect characters with backslashes. This works for every character except newlines; for a newline, putting it inside single quotes (or double quotes) is the only solution. If you want to minimize the number of backslashes to present the quoted name to the user, you may restrict it to places where the backslash is needed; however, the more backslashes you omit, the more you risk forgetting one that's needed. Letters, digits and non-ASCII characters are always ok¹. Quote whitespace and punctuation whenever you're unsure.

With a typical shell (ksh, bash or zsh), you need to quote the following characters in at least some circumstances.

• Whitespace (space, tab, newline — remembering that newlines can't be quoted with a backslash).
• ! — history expansion.
• " — shell syntax.
• # — comment start when preceded by whitespace; zsh wildcards.
• $ — shell syntax.
• & — shell syntax.
• ' — shell syntax.
• ( — even in the middle of a word: ksh extended globs (also available in bash and zsh); zsh wildcards.
• ) (see ()
• * — sh wildcard.
• , — only inside brace expansion.
• ; — shell syntax.
• < — shell syntax.
• = — in zsh, when it's at the beginning of a file name (filename expansion with PATH lookup).
• > — shell syntax.
• ? — sh wildcard.
• [ — sh wildcard.
• \ — shell syntax.
• ] — you may get away with leaving it unquoted.
• ^ — history expansion; zsh wildcard.
• ` — shell syntax.
• { — brace expansion.
• | — shell syntax.
• } — needs to be escaped in zsh, other shells are more lenient when there's no matching opening brace.
• ~ — home directory expansion when it's at the beginning of a file name; zsh wildcard; always safe when it's the last character.

A few more characters can require special handling sometimes:

• - isn't special for the shell, but when it's at the beginning of a command argument, it indicates an option. It can't be protected with quotes since the special handling is in the command, not in the shell. To protect a file name that begins with -, you can put ./ before it — this way it's still the same file, but the argument doesn't begin with - anymore.
• . isn't special in itself, but dot files are excluded from * globs by default.
• : isn't special for the shell, but some commands parse it specially, e.g. to indicate a remote file (hostname:filename). Consult the documentation of the command to see how to cope with file nams containing colons.

¹ _{Unless the user has configured alternate history expansion characters. Some shells allow this. This is another reason to use single quotes rather than backslashes.}

Comments

[iBug]

Newlines, at least LF only, can be quoted with a backslash in some shells.

[Gilles 'SO- stop being evil']

@iBug No. In any sh-like shell, backslash+newline expands to an empty string, not to a newline.

[slashmais]

This is a good answer, thank you. Is there a link to where this information is 'officially' available?

[Gilles 'SO- stop being evil']

@slashmais What information? The syntax of the shell language is “officially” available in various manuals and specifications, but going from there to finding out exactly which characters need quoting where isn't completely straightforward.

[slashmais]

I also found this: tecmint.com/manage-linux-filenames-with-special-characters - it is not as compact as your list, but has plenty examples. (by-the-by: "What information?" is not a cool response when the topic here concerns the characters)