How do I "cat and follow" a file?

Question

A file is being sequentially downloaded by wget.

If I start unpacking it with cat myfile.tar.bz2 | tar -xj, it may unpack correctly or fail with "Unexpected EOF", depending on what is faster.

How to "cat and follow" a file, i.e. output content of the file to stdout, but don't exit on EOF, instead keep subsribed to that file and continue outputting new portions of the data, exiting only if the file is closed by writer and not re-opened within N seconds.

---

I've created a script cat_and_follow based on @arielCo's answer that also terminates the tail when the file is not being opened for writing anymore.

Solution 1

tail +1f file

I tested it on Ubuntu with the LibreOffice source tarball while wget was downloading it:

tail +1f libreoffice-4.2.5.2.tar.xz | tar -tvJf -

It also works on Solaris 10, RHEL3, AIX 5 and Busybox 1.22.1 in my Android phone (use tail +1 -f file with Busybox).

Solution 2

The problem is that cat is not aware that the file is still being appended. As soon as cat encounters the (current) end of the file it exits.

You have to make wget write to a pipe (or FIFO) in order to avoid this.

wget -O - http://... | tar -xjf -

Solution 3

To read and follow a file from the beginning until interrupted:

tail -fn +1 file

To demonstrate that, try this (assuming Bash with GNU Coreutils):

(while true; do printf . >> /tmp/file; sleep 1; done)&
tail -fn +1 /tmp/file  # (Ctrl-C to interrupt, of course, or otherwise kill it.)
kill %  # Kills the while-loop.

(Note: The +1f mentioned by others is interpreted as a filename, at least in the GNU tail command.)

The above works for a single file. Concatenation of multiple files would not be able to follow all of them deterministically, without hanging on the first. To ‘cat and follow’, following only the last file, one can use process substitution. Here's another demonstration:

printf file1 > /tmp/file1; printf file2 > /tmp/file2
(while true; do printf . | tee -a /tmp/file{1,2} > /dev/null; sleep 1; done)&
cat /tmp/file1 <(tail -fn +1 /tmp/file2)  # (Interrupt or kill it.)
kill %  # Kills the while-loop.

Comments

[Kevin]

Or curl, which outputs to stdout by default. And (1) -f - is redundant, tar reads from stdin by default, and (2) most tars can detect the compression automatically, so the j is often unnecessary. curl http://... | tar x

[Vi.]

This will download without saving the unpacked file. Also it will hinder continuing the download in case of bad network.

[Vi.]

I didn't expect that tail works with binray files...

[Vi.]

Created a script that does scanning /proc/../fd and auto-termination.

[Rob C]

@Vi. You can save the file by using tee like this: curl http://… | tee ….tbz | tar -xj, but resuming the download gets more complicated than just invoking the same command again.

[Schism]

@Vi. It's great that you solved your problem. Can you share your solution?

[Vi.]

@Schism, It's already shared on Github Gists. See the link in the edited question.

[Barmar]

You don't need GNU tail, I believe the -f option is standard. However, it won't terminate automatically when the download is finished.

[Aaron Fox]

@Barmar: You're right - I tested it on Solaris, adjusting the switches for compatibility. Edited the answer accordingly.

[blueFast]

In Ubuntu tail +1f does not work. You have to do tail -n +1

[mwfearnley]

Agreed, I've just tried Ubuntu 16.04, and it tried to read +1 as a file. So it seems tail -n+1 -f is necessary.

[2xsaiko]

This worked for me while tail +1f file didn't.

[Joseph K. Strauss]

This answer is much better than tail +1f when combining it with other options such as --pid

[Gabriel Staples]

What does the +1 do?

[Aaron Fox]

@GabrielStaples +n = start printing at the n-th line (or byte). linux.die.net/man/1/tail