Command line 'buffer'

Question

I wanted to tail the last 100 lines of a file to the same file, but the command tail -n 100 file > file doesn't work, I assume because the stdout gets written to the file 'live', before everything was read from the original file.

Is there some way to pipe the output to something, that then keeps it until all 100 lines are there, and then outputs it to the file? Or just another way to shorten the file in this way?

Solution 1

sponge from moreutils is good for this. It will:

soak up standard input and write to a file

You use it like this:

tail -n 100 file | sponge file

to get exactly the effect you want.

Solution 2

Of course, the moment I finally ask the question on SE, the answer comes to me. I think less does what I need, so I just write:

tail -n 100 file | less > file

Solution 3

In:

tail -n 100 file > file

The shell forks a process, opens file for writing in it, with truncation (that is making it an empty file) and then executes tail in that process. To open file without truncation, you can use the <> redirection operator instead:

tail -n 100 file 1<> file

The problem though is that there will be no truncation at all. That is, file will be overridden with its last 100 lines, but after those 100 lines, what was originally in the file will still be there. So you'd need to call another command to do the truncating after tail has finished.

{ tail -n 100 file; perl -e 'truncate STDOUT, tell STDOUT'; } 1<> file

Comments

[MadTux]

I wouldn't mark it as a duplicate, as my problem has been wonderfully solved by sponge. Still, the link is good for people (like me) who want to know more about it.

[MadTux]

Much nicer than less :) Also, thanks for introducing me to moreutils. That looks very useful. I think I'll accept this one.

[Michael Homer]

It's actually the redirection (>) that truncates the file, rather than the command you run - this won't (or at least shouldn't!) work. The shell opens and empties the destination before it even starts the commands you give. See the (slightly obtuse) description of output redirection in POSIX.

[Michael Homer]

Unless any lines contain more than one word or are blank.

[Michael Homer]

@illuminÉ: That is a neat point. So I should say that this does work very occasionally, but nondeterministically.

[Internal Server Error]

@MichaelHomer Re this: really just an edge case, just thought of sharing it as it was insightful in context! Cheers!

[MadTux]

OK, thanks for telling me. I think I'll downvote my own answer x) EDIT: I can't. Hmmm.

[Lonnie Best]

Sponge; I love analogy; great name!