Why won't `tail -f` follow my syslog when running live?

Question

I want to monitor /var/log/syslog for any changes in realtime (or within-a-few-seconds-of-time), but tail -f won't update with any new changes.

I'm running Ubuntu-based Linux Mint 17 XFCE live now, but this hasn't worked on live iso's of Ubuntu, Xubuntu, or Linux Mint Mate either.

I've tried these:

$ tail -f /var/log/syslog
$ tail -f --retry -s 1 /var/log/syslog
$ tail --follow=name /var/log/syslog
$ tail --follow=name --retry /var/log/syslog
$ tail --follow=name --retry -s 1 /var/log/syslog

But it only initially outputs the last few lines of the file, then no updates when the file grows (when, for example, trying to mount an empty file gives about 15 lines of errors).

Actually, even trying to follow a test file in my home folder doesn't seem to work, running tail -f testfile and then (in another terminal):
$ echo "new stuff" >> testfile
$ echo "new stuff2" >> testfile
$ echo "3" >> testfile
doesn't result in any updates to tail either...
But if I put testfile in /tmp(mounted on a tmpfs) then it does follow the file's changes.

Why won't tail follow?

Is there something weird about running live, or overlayfs that cripples tail -f? And any suggestions how to follow the log? (xwatch works ok, anything better or in terminal?)

---

I've tried running strace tail -f -s 1 testfile and here are the last couple lines of output, after it write(1,'s the existing couple lines of testfile:

write(1, "new1\n", 5new1
)                   = 5
fstat64(3, {st_mode=S_IFREG|0644, st_size=22, ...}) = 0
fstatfs64(3, 84, {f_type=0x1021994, f_bsize=4096, f_blocks=968776, f_bfree=461437, f_bavail=461437, f_files=203469, f_ffree=190635, f_fsid={0, 0}, f_namelen=255, f_frsize=4096, f_flags=1056}) = 0
inotify_init()                          = 4
inotify_add_watch(4, "testfile", IN_MODIFY|IN_ATTRIB|IN_DELETE_SELF|IN_MOVE_SELF) = 1
fstat64(3, {st_mode=S_IFREG|0644, st_size=22, ...}) = 0
read(4, 

Solution

tail uses inotify which does not work with overlayfs See this bug report and this discussion. @Xen2050 pointed out the ---disable-inotify switch to tail See this workaround

You could use apt-src to install the coreutils source, and recompile tail with -UHAVE_INOTIFY

Comments

[goo]

How long do you wait before declaring a failure? Have you read man tail? The --sleep-interval=N parameter might help. Or you could strace tail -f /var/log/syslog and see where tail is waiting.

[Xen2050]

Thanks for the input, I'll try strace asap. I did check man tail, and the -s in my trials is an alias for --sleep-interval. Not at all sure how to interpret the strace results, but I'll paste the last few lines after it prints the existing file contents, last line is a read(4,. Beginning to think it is a bug, or just incompatible with overlayfs. I wait a few minutes after an echo new >> testfile, nothing happens

[goo]

tail uses inotify to watch the files. Does inotify work with overlayfs? It seems not to. See bugs.launchpad.net/ubuntu/+source/linux/+bug/882147 (sorry about doubled comment - while I was finding out about tail, inotify and overlayfs, my earlier comment timed out). Also see help.lockergnome.com/linux/…

[Rinzwind]

@waltinator it does not :) (and you can remove comments ;) )

[Xen2050]

@waltinator I just found that first bug too, from searching "Does inotify work with overlayfs". And the "undocumented" tail workaround ---disable-inotify that makes tail work "like it used to" (apparently it started ignoring --sleep-interval around 2010 (linked from the first bug report). If you added the tail ---disable-inotify... (from comment 30) that would be a perfect answer. Thanks!