Default field separator for awk

Question

Is the default separator only space for awk?

Solution 1

Here's a pragmatic summary that applies to all major Awk implementations:

• GNU Awk (gawk) - the default awk in some Linux distros
• Mawk (mawk) - the default awk in some Linux distros (e.g., earlier versions of Ubuntu ^{crysman reports that version 19.04 now comes with GNU Awk - see his comment below.})
• BSD Awk - a.k.a. BWK Awk - the default awk on BSD-like platforms, including OSX

^{On Linux, awk -W version will tell you which implementation the default awk is.
BSD Awk only understands awk --version (which GNU Awk understands in addition to awk -W version).}

Recent versions of all these implementations follow the POSIX standard with respect to field separators^[1] (but not record separators).

Glossary:

• RS is the input-record separator, which describes how the input is broken into records:

  • The POSIX-mandated default value is a newline, also referred to as \n below; that is, input is broken into lines by default.
  • On awk's command line, RS can be specified as -v RS=<sep>.
  • POSIX restricts RS to a literal, single-character value, but GNU Awk and Mawk support multi-character values that may be extended regular expressions (BSD Awk does not support that).

• FS is the input-field separator, which describes how each record is split into fields; it may be an extended regular expression.

  • On awk's command line, FS can be specified as -F <sep> (or -v FS=<sep>).
  • The POSIX-mandated default value is formally a space (0x20), but that space is not literally interpreted as the (only) separator, but has special meaning; see below.

---

By default:

• any run of spaces and/or tabs and/or newlines is treated as a field separator
• with leading and trailing runs ignored.

^{The POSIX spec. uses the abstraction <blank> for spaces and tabs, which is true for all locales, but could comprise additional characters in specific locales - I don't know if any such locales exist.}

Note that with the default input-record separator (RS), \n, newlines typically do not enter the picture as field separators, because no record itself contains \n in that case.

Newlines as field separators do come into play, however:

• When RS is set to a value that results in records themselves containing \n instances (such as when RS is set to the empty string; see below).
• Generally, when the split() function is used to split a string into array elements without an explicit-field separator argument.
  • Even though the input records won't contain \n instances in case the default RS is in effect, the split() function when invoked without an explicit field-separator argument on a multi-line string from a different source (e.g., a variable passed via the -v option or as a pseudo-filename) always treats \n as a field separator.

---

Important NON-default considerations:

• Assigning the empty string to RS has special meaning: it reads the input in paragraph mode, meaning that the input is broken into records by runs of non-empty lines, with leading and trailing runs of empty lines ignored.

• When you assign anything other than a literal space to FS, the interpretation of FS changes fundamentally:

  • A single character or each character from a specified character set is recognized individually as a field separator - not runs of it, as with the default.
    • For instance, setting FS to [ ] - even though it effectively amounts to a single space - causes every individual space instance in each record to be treated as a field separator.
    • To recognize runs, the regex quantifier (duplication symbol) + must be used; e.g., [\t]+ would recognize runs of tabs as a single separator.
  • Leading and trailing separators are NOT ignored, and, instead, separate empty fields.
  • Setting FS to the empty string means that each character of a record is its own field.
• As mandated by POSIX, if RS is set to the empty string (paragraph mode), newlines (\n) are also considered field separators, irrespective of the value of FS.

---

<sup>[1] Unfortunately, GNU Awk up to at least version 4.1.3 complies with an obsolete POSIX standard with respect to field separators when you use the option to enforce POSIX compliance, -P (--posix): with that option in effect and RS set to a non-empty value, newlines (\n instances) are NOT recognized as field separators. The GNU Awk manual spells out the obsolete behavior (but neglects to mention that it doesn't apply when RS is set to the empty string). The POSIX standard changed in 2008 (see comments) to also consider newlines field separators when FS has its default value - as GNU Awk has always done without -P (--posix).
Here are 2 commands that verify the behavior described above:
* With -P in effect and RS set to the empty string, \n is still treated as a field separator:
gawk -P -F' ' -v RS='' '{ printf "<%s>, <%s>\n", $1, $2 }' <<< $'a\nb'
* With -P in effect and a non-empty RS, \n is NOT treated as a field separator - this is the obsolete behavior:
gawk -P -F' ' -v RS='|' '{ printf "<%s>, <%s>\n", $1, $2 }' <<< $'a\nb'
A fix is coming, according to the GNU Awk maintainers; expect it in version 4.2 (no time frame given).
(Tip of the hat to @JohnKugelman and @EdMorton for their help.)</sup>

Solution 2

The question the default delimiter is only space for awk? is ambiguous but I'll try to answer both of the questions you might be asking.

The default value of the FS variable (which holds the field separator that tells awk how to separate records into fields as it reads them) is a single space character.

The thing that awk uses to separate records into fields is a "field separator" which is a regular expression with some additional functionality that only applies when the field separator is a single blank character. That additional functionality is that:

1. Leading and trailing white space is ignored during field splitting.
2. Fields are separated at chains of contiguous space characters which includes blanks, tabs and newlines.
3. If you want to use a literal blank character as a field separator you must specify it as [ ] instead of just a standalone literal blank char like you could in a regexp.

In addition to field separators being used to split records into fields as the input is read they are used in some other contexts, e.g. the 3rd arg for split(), so it's important for you to know which contexts require a string or a regexp or a fieldsep and the man page clearly specifies each.

Among other things, the above explains this:

$ echo ' a b c ' | awk '{printf "%d: <%s> <%s> <%s>\n", NF, $1, $2, $3}'
3: <a> <b> <c>
$ echo ' a b c ' | awk -F' ' '{printf "%d: <%s> <%s> <%s>\n", NF, $1, $2, $3}'
3: <a> <b> <c>
$ echo ' a b c ' | awk -F'[ ]' '{printf "%d: <%s> <%s> <%s>\n", NF, $1, $2, $3}'                              
5: <> <a> <b>

so if you don't understand why the first 2 produce the same output but the last is different, please ask.

Solution 3

Let's take a look at the GNU awk man page:

FS — The input field separator, a space by default. See Fields, above.

To the Fields section!

As each input record is read, gawk splits the record into fields, using the value of the FS variable as the field separator. If FS is a single character, fields are separated by that character. If FS is the null string, then each individual character becomes a separate field. Otherwise, FS is expected to be a full regular expression. In the special case that FS is a single space, fields are separated by runs of spaces and/or tabs and/or newlines.

Comments

[Thor]

The default field delimiter or field separator (FS) is [ \t]+, i.e. one or more space and tab characters.

[Ed Morton]

@Thor no, it's not. See the man page.

[Thor]

@EdMorton: Right, I missed newlines, i.e. FS='[ \t\n]+. But this only has an effect when RS does not include newlines.

[Ed Morton]

@Thor not exactly since even if you have an RS that includes newlines the default FS will have an effect if you construct a string containing newlines and you do split(string,arr).

[Timo]

Good question, not stupid at all

[Ed Morton]

The above is not correct even for gawk. In a POSIX awk a newline IS a separator when FS has it's default value so using --posix in gawk won't change that behavior.

[mklement0]

Please don't confuse "blank" with "space". "space" is the actual space character (0x20), whereas "blank" is a potentially locale-specific abstraction: "In the POSIX locale, only the <space> and <tab> shall be included. In a locale definition file, the <space> and <tab> are automatically included in this class." (I can see no umbrella term in the POSIX spec that covers both "blank" and newlines.)

[Lin Ma]

Hi John, a bit lost in your reply. Does it mean only space, or both space/Tab are used as default delimiter?

[Lin Ma]

Thanks mklement0, I read the reply from John as well and it seems only space is default delimiter? However you mentioned both space and Tab? Please feel free to correct me if I am wrong. :)

[mklement0]

Otherwise, a great answer.

[mklement0]

In short: While a space is formally the default FS value, it stands for space, tabs, and newlines. I've updated my answer to clarify.

[mklement0]

Just to complement this answer: even though the quotes are from the GNU Awk manual page, they also apply to the other Awk implementation that some Linux distros come with by default, Mawk (mawk; e.g., on Ubuntu) - and they also apply to BSD Awk, as found on BSD-like platforms, including OSX.

[Ed Morton]

The POSIX standard changed and gawk is supporting the older version. See the 2004 standard (pubs.opengroup.org/onlinepubs/009695399/utilities/awk.html) which states a field is a string of non- <blank>s vs the 2013 standard (pubs.opengroup.org/onlinepubs/9699919799/utilities/awk.html‌​) which states a field is a string of non- <blank> non- <newline> characters. You should email [email protected] about that.

[mklement0]

@EdMorton: Turns out I misread the POSIX spec and you were correct: the spec did change as you describe in POSIX.1-2008 (SUS v4). Per your suggestion (thanks) I've emailed [email protected] and have since heard back that a fix is coming in the "next major release", but I'm unclear on what version that is.

[crysman]

@mklement0 it seems Ubuntu 19.04 comes with both gawk and mawk, here is where you find it: releases.ubuntu.com/19.04/ubuntu-19.04-desktop-amd64.manifes‌​t . BUT, indeed, gawk is default: ❱ which -a "awk" /usr/bin/awk ❱ file /usr/bin/awk /usr/bin/awk: symbolic link to /etc/alternatives/awk ❱ file /etc/alternatives/awk /etc/alternatives/awk: symbolic link to /usr/bin/gawk

[mklement0]

Thanks, @crysman - I've updated the answer to point to your comment.