Do progress reports/logging information belong on stderr or stdout?

Question

Is there an official POSIX, GNU, or other guideline on where progress reports and logging information (things like "Doing foo; foo done") should be printed? Personally, I tend to write them to stderr so I can redirect stdout and get only the program's actual output. I was recently told that this is not good practice since progress reports aren't actually errors and only error messages should be printed to stderr.

Both positions make sense, and of course you can choose one or the other depending on the details of what you are doing, but I would like to know if there's a commonly accepted standard for this. I haven't been able to find any specific rules in POSIX, the GNU coding standards, or any other such widely accepted lists of best practices.

We have a few similar questions, but they don't address this exact issue:

• When to use redirection to stderr in shell scripts: The accepted answer suggests what I tend to do, keep the program's final output on stdout and anything else to stderr. However, this is just presented as a user's opinion, albeit supported by arguments.

• Should the usage message go to stderr or stdout?: This is specific to help messages but cites the GNU coding standard. This is the sort of thing I'm looking for, just not restricted to help messages only.

So, are there any official rules on where progress reports and other informative messages (which aren't part of the program's actual output) should be printed?

Solution 1

Posix defines the standard streams thus:

At program start-up, three streams shall be predefined and need not be opened explicitly: standard input (for reading conventional input), standard output (for writing conventional output), and standard error (for writing diagnostic output). When opened, the standard error stream is not fully buffered; the standard input and standard output streams are fully buffered if and only if the stream can be determined not to refer to an interactive device.

The GNU C Library describes the standard streams similarly:

Variable: FILE * stdout
The standard output stream, which is used for normal output from the program.

Variable: FILE * stderr
The standard error stream, which is used for error messages and diagnostics issued by the program.

Thus, standard definitions have little guidance for stream usage beyond “conventional/normal output” and “diagnostic/error output.” In practice, it’s common to redirect either or both of these streams to files and pipelines, where progress indicators will be a problem. Some systems even monitor stderr for output and consider it a sign of problems. Purely auxiliary progress information is therefore problematic on either stream.

Instead of sending progress indicators unconditionally to either standard stream, it’s important to recognize that progress output is only appropriate for interactive streams. With that in mind, I recommend writing progress counters only after checking whether the stream is interactive (e.g., with isatty()) or when explicitly enabled by a command-line option. That’s especially important for progress meters that rely on terminal update behavior to make sense, like %-complete bars.

For certain very simple progress messages (“Starting X” ... “Done with X”) it’s more reasonable to include the output even for non-interactive streams. In that case, consider how users might interact with the streams, like searching with grep or paging with less or monitoring with tail -f. If it makes sense to see the progress messages in those contexts, they will be much easier to consume from stdout.

Solution 2

POSIX defines standard error as

for writing diagnostic output

This doesn't limit its use to error messages only. I would consider progress information as diagnostic output, so it belongs on standard error.

Solution 3

By the principle of exclusion, it can only go to stderr. Yes, I know you asked about an official specification, which I cannot present you beyond the link to the POSIX specification, given by Stephen Kitt, which states that stderr is for diagnostic purposes.

The more important point is that stdin and stdout have a function that disallows printing progress reports to stdout - they of course form the sequence of pipes which in Unix shell commands is not just a side-effect, but the very core of the powerful pipelining approach.

So. Nothing except the real "payload" of your program belongs on stdout. If your program has no output, then nothing should go to stdout. This leaves stderr for everything else, including progress reports.

Granted, this leaves a hole - it would probably be nice to have a "stdfluff" or something like that which is for neither output nor errors but progress reports, debugging and somesuch. In fact, nothing keeps you from doing that, i.e., you could print your progress to file descriptor 3. Example:

$ perl -e 'open($fd, ">", "/dev/fd/3"); print $fd "hello\n"'

This produces no output. (*)

$ perl -e 'open($fd, ">", "/dev/fd/3"); print $fd "hello\n"'  3>&1
hello

This prints to fd-3, which is redirected to stdout.

(*) The first example produces no output but is still a bit far-fetched; the open fails and $! would contain no such file or directory; just take this as an example, it is of course not ment to be used like this in earnest. In an actual program, if you wanted to go this route, you could test whether /dev/fd/3 is usable and take this as a hint of whether to activate your progress reports; you'd have to do that pretty early so you don't get confused by your own opens for real files and such...

Solution 4

POSIX is slightly more concrete about "diagnostic information" in Shell and Utilities, 1.4: Utility Description Defaults (emphasis mine):

STDERR

The STDERR section describes the standard error output of the utility. Only those messages that are purposely sent by the utility are described. Use of a terminal for standard error may cause any of the standard utilities that write standard error output to stop when used in the background. For this reason, applications should not use interactive features in scripts to be placed in the background.

The format of diagnostic messages for most utilities is unspecified, but the language and cultural conventions of diagnostic and informative messages whose format is unspecified by this volume of POSIX.1-2008 should be affected by the setting of LC_MESSAGES and [XSI] [Option Start] NLSPATH. [Option End]

The specified standard error output of standard utilities shall not depend on the existence or value of the environment variables defined in this volume of POSIX.1-2008, except as provided by this volume of POSIX.1-2008.

Default Behavior: When this section is listed as "The standard error shall be used only for diagnostic messages.", it means that, unless otherwise stated, the diagnostic messages shall be sent to the standard error only when the exit status indicates that an error occurred and the utility is used as described by this volume of POSIX.1-2008.

When this section is listed as "Not used.", it means that the standard error shall not be used when the utility is used as described in this volume of POSIX.1-2008.

IANASL, but I interpret that to mean that stderr will have output only if the utility will return an error exit code. Since this should not be the normal course of events for successful execution, no progress information should be printed by a POSIX utility unless an error occurs (unless, of course, otherwise specified, etc.).

View more solutions

Comments

[Admin]

Printing spinners and the like to stdout together with the results is a safe way to make the results useless. If you ever need to pipe the results to some other program, said program would need to separate the results from the spinners. Also, if you redirect the output to a file you won't be able to see the spinners. IMHO.

[Admin]

@SatoKatsura yeah, that's my opinion as well and that's why I print such to stderr. The CTO of the company I work for, however, feels that printing to stderr is an indication that something went wrong. I made the, rather bold, claim that the POSIX Way® is printing to stderr and he called me out on it. Given that he has 20 odd years of experience on me, I would like to see if I can find some sort of "official" guideline.

[Admin]

same as Sato Katsura said, stdout is actually a safe and right way to pringting spinners and other informative messages, but as many programmer say 'Silence is golden. Output nothing if everything is fine.' so in fact, stderr is always used to do that because of the vague definition and also stdout may break the pipe sequence.for official guide

[Admin]

@Seven I think you misunderstood; Sato is saying that using stdout is unsafe ("safe way to make the results useless").

[Admin]

How I wish that we had a 'meta' file descriptor

[Admin]

One possible one-size-fits-all solution would be to only use stderr to report error messages, except when a --verbose flag is used, at which point progress reports are included as well.

[PM 2Ring]

FWIW, in Python, stdout is line-buffered by default whereas stderr is unbuffered, so stderr is the natural choice for writing progress text / bars / spinners that don't contain a newline. (If you write such text on stdout you need to clutter up your progress output calls with stdout.flush() to make it visible).

[Stephen Kitt]

@PM2Ring that's not specific to Python, POSIX defines the streams that way.

[terdon]

Sorry, but this isn't answering the question. I specifically asked for official guidelines, not opinion. In any case, even if I had asked for opinion, you are answering a different question. I am not talking about --help or help messages at all. Please re-read the question.

[Random832]

You mentioned "usage message", I thought it was part of your question rather than just the title of the other question you linked. I don't understand why you think "official guidelines" exist for this. Who would have the authority to make them?

[terdon]

I don't know that they do, that's why I ask. And as for official, the POSIX standard has specifications for various minute details of how a program should behave. Including, as Stephen pointed out in his answer, a vague suggestion for what should go to stderr. The GNU coding standard could also have something similar. Basically, any group that is active in and produces code for the *nix ecosystem might have a standard I'd be interested in.

[Artashes Aghajanyan]

@PM2Ring: When writing to stdout, you need to flush even if your text does contain a newline if you want your progress reports to actually show up in real time when redirected.

[Random832]

@terdon would you be okay with examples of what existing programs do? curl, wget, and pv all use stderr, with wget and pv making it conditional on it being a tty and curl making it conditional on stdout not being the tty.

[PM 2Ring]

@Hurkyl Ah, good point. I wasn't considering redirection.

[Random832]

Also, it sounds like your CTO thinks that a calling program should react to the presence of data on stderr as meaning that an error has occurred, rather than looking at the exit status. This is definitely bad practice (and also harder to write than checking the exit status) and might be a better angle to convince him from.

[terdon]

Nah, he knows his stuff. It's a question of style rather than function. That's why I would really like to have some sort of "official" rule/convention to point to. And that's why examples are helpful (although I already know them) but not really what I'm after.

[Wtower]

I would agree, but please also mind that several tools like Ansible fail a task if the relevant programme has output in stderr and they need special handling.

[Charles Duffy]

@Wtower, eh? No. Ansible swallows stderr text if exit status is 0. That claim is simply false.

[Wtower]

@CharlesDuffy no sweat, my wrong, thanks for clearing this out.

[Stephen Kitt]

I understand this as describing the meaning of the "STDERR" section given in the specifications of each POSIX utility, not as a general definition for standard error usage. I don't think terdon is writing a standard POSIX utility here ;-).

[muru]

@StephenKitt neither do I. But he's debating with his CTO, who says that stderr output is an indication of something going wrong, and the standard does support his claim as the default behaviour.

[Stephen Kitt]

It only supports the claim as the default behaviour for POSIX utilities, and even then only in specific cases (utilities which use the phrase mentioned). This part of POSIX isn't normative (AIUI), it's a template: it tells you how to read the utility specifications, it doesn't specify the utilities' behaviour. Specifically, it defines the meanings of the phrases "The standard error shall be used only for diagnostic messages." and "Not used." in the "STDERR" sections of utility specifications. Each utility specifies its behaviour, and it can use those phrases as shorthand.

[muru]

@StephenKitt which doesn't change my point. POSIX utilities either a) don't output to stderr, b) use it only for diagnostic messages if something goes wrong, c) use it only for diagnostic messages even if nothing goes wrong and d) use it for something else. I said that they only use it for diagnostic messages if something goes wrong (a, b), unless otherwise stated (c, d). Now, my claim is that this is the default, which it clearly is, since that's why it's template.

[Stephen Kitt]

Well, I'd say the "unless otherwise specified" is a rather significant caveat: a POSIX utility could very well specify that it outputs progress information on its standard error, and it would be standards-compliant. You're right, it is interesting that the "default behaviour" (which still requires a specific mention in the relevant section) is to only use standard error if the exit code also indicates an error; but it's not limiting.

[muru]

@StephenKitt of course. We know of utilities which output to stderr without being in an abnormal state - dd has to output stats on completion, rm and find do when they have to prompt (this could be taken as progress information).

[terdon]

Thanks, the GNU guideline is the sort of thing I'm after. However, I realize my question was a little misleading. When I mentioned "progress reports" I was thinking of both things like N% done and things like doing foo and foo done. The latter should always be kept since they are used for debugging when redirected to a file. So your suggestion of checking whether the stream is interactive isn't applicable (though generally a very good idea).

[psmears]

@CharlesDuffy: I'm sure you're right about Ansible, but wouldn't you need echo hello 1>&2 rather than echo hello >stderr to write "hello" to stderr? Won't the latter write "hello" to a file called "stderr" in the current directory?

[Charles Duffy]

@psmears, you're quite right. ansible -m shell --args "echo hello >&2" localhost is indeed the correct test.

[Bradd Szonye]

Updated answer to incorporate clarifications above.

[Charles Duffy]

If it were the default, then it would be the behavior in question if not explicitly otherwise specified, as opposed to the behavior in question only if a utility's specification uses the words "The standard error shall be used only for diagnostic messages" -- itself an explicit specification.

[Peter Mortensen]

What is fd-3? The third floppy disk?

[AnoE]

file descriptor 3, @PeterMortensen

[vijay]

I like your suggestion of writing to /dev/tty directly rather than stderr (if it isatty), which is what I'm researching now, which led me to this question.