why it is 2>&1 and not 2>>&1 to append to a log file

Question

I am redirecting STDOUT and STDERR to a single file, always growing, so I use always use "append" redirection; this is >>

my command is command >> logfile 2>&1

And it works.

But the STDERR redirection has a single >, which I was using to "create" the file, erasing the previous one, as in command > outlog 2> errlog

Why it does not erase the log file in this case?

Solution 1

When you redirect something to &number, you are not opening a new file at all; you're reusing an already open file along with whatever mode it was opened.

The numbers refer to "open file" handles (file descriptors). So there is no technical difference between how >& and >>& (and indeed <&) would work – they all just mean "clone the existing file descriptor using dup()".

That is, 2>&1 indicates that file descriptor #1 (which you previously opened for appending using >>logfile) is cloned into number #2. And yes, 2<&1 works identically.

Side technical note: Appending vs truncating is not an explicit action done by the shell; it's actually a mode the shell specifies when opening the file, and the rest is performed by the OS itself. For example, when you use > the shell doesn't manually erase the old contents, it just adds O_TRUNC when calling open(). Therefore, when open() isn't called at all, the previous mode remains unchanged.

Solution 2

The sequence command >> logfile 2>&1 has two redirection stages:

• command >> logfile will append to the logfile
• 2>&1 will redirect stderr to stdout (which itself appends to the logfile)

So command >> logfile 2>&1 will not truncate the log file, while command >>logfile 2>logfile would.

Comments

[gronostaj]

Welcome to Super User and congrats on an interesting first question!

[Daniel Wagner]

The tokens aren't split up the way you think they are. That is, it's not "> for redirect to file" and "&1 for which file to redirect to", it's ">& for redirect to file descriptor (which is not the same thing as a file)" and "1 for which descriptor to redirect to". So it's a completely different operation, and you shouldn't a priori expect to be able to transfer any intuition from > or >> to >&. (This is just a first approximation. In the second approximation, there are some implementation details shared between the three operators, so you can transfer some knowledge...)

[David Richerby]

@DanielWagner Please don't answer in comments.

[barlop]

Is O_TRUNC something you run into when doing windows programming in C?

[user1686]

@barlop: Yes; not exactly, but quite similar. If you develop on raw Win32 API, the CreateFile() function has TRUNCATE_EXISTING, which means the same thing as POSIX open()'s O_TRUNC, and you have DuplicateHandle() which is like dup(), even if Windows' stdin/stdout are a bit weirder and more special-cased. (If you develop on Mingw, you just get the same POSIX open(). And most higher level I/O libraries, like C stdio or Python io, have modes such as "w" "a" "r+" which directly translate to the same thing.)

[Zereges]

What about command >>logfile 2>>logfile?

[Tonny]

@Zereges Probably gives an error because you try to open logfile for writing twice. One of the 2 "opens" should fail with an error along the lines of "file already open (or locked) by another process"

[user1686]

No, files on Unix-like systems are not locked merely by being open. (For that matter, e.g. Linux doesn't support mandatory locking at all.)

[Oliv]

@grawity, your explanation suggests that command 2>&1 >> logfile will overwrite the file, which is not the case, could you elaborate that? In my experiment error output went to console and standard output went to the file.

[user1686]

@Oliv: Does it? You're not using any "overwrite" redirection operator, you're just using the same "duplicate fd" and "append to file".

[user1686]

Note that "cloning" here is not "aliasing". Once 2>&1 clones fd 1 to fd 2, they're still two independent file descriptors that merely reference the same "open file". Further redirections of fd 1 do not affect fd 2.

[Oliv]

I think what it does is that it first redirects error output to console and then standard output to console. So the order matters.

[Peter Cordes]

Fun fact: O_APPEND is a special flag that's separate from just omitting O_TRUNC. It means any write automatically happens at the end of the file, even if other writers made the file longer since your last write. i.e. every write() becomes like an atomic combination of lseek(SEEK_END) + write(). You can open a file for overwrite without truncation using for example dd of=foo conv=notrunc

[Peter Cordes]

@Zereges: You'd get 2 separate file descriptors (not duplicates of each other) each with their own file position. But they're both opened in O_APPEND mode, so every write is like an atomic lseek(SEEK_END) + write(). So I think it would Just Work with the same end result as normal dup2() but probably slightly less efficient.

[Attie]

Also, note that order is important... 2>&1 >> logfile will redirect stderr to the terminal, and then subsequently redirect the application's stdout to logfile.