Tail log file on multiple machines over ssh

Question

I'm trying to tail a log file on multiple remote machines and forward the output to my local workstation. I want connections to close when pressing Ctrl-C.

At the moment I have the following function that almost works as intended.

function dogfight_tail() {
 logfile=/var/log/server.log
 pids=""
 for box in 02 03; do
   ssh server-$box tail -f $logfile | grep $1 &
   pids="$pids $!"
 done
 trap 'kill -9 $pids' SIGINT
 trap  wait
}

The connections close and I receive the output from tail. BUT, there is some kind of buffering going on because the output come in batches.

And here's the fun part…

I can see the same buffering behaviour when executing the following and append "test" to the file /var/log/server.log on the remote machines 4-5 times…

ssh server-01 "tail -f /var/log/server.log | grep test"

…and found two ways of disabling it…

1. Add -t flag to ssh.

   ssh -t server-01 "tail -f /var/log/server.log | grep test"
   

2. Remove quotation from the remote command.

   ssh server-01 tail -f /var/log/server.log | grep test
   

However, neither of these approaches work for the function that execute on multiple machines mentioned above.

I have tried dsh, which have the same buffering behaviour when executing.

dsh -m server-01,server-02 -c "tail -f /var/log/server.log | grep test"

Same here, if I remove the quotation, the buffering goes away and everything works fine.

dsh -m server-01,server-02 -c tail -f /var/log/server.log | grep test

Also tried parallel-ssh which works exactly the same as dsh. Can somebody explain what's going on here?

How do I fix this problem? Would be ideal to go with straight ssh if possible.

P.S. I do not want to use multitail or similar since I want to be able to execute arbitrary commands.

Solution 1

What you see is effect of a standard stdout buffer in grep provided by Glibc. The best solution is to disable it by using --line-buffered (GNU grep, I'm not sure what other implementations might support it or something similar).

As for why this only happens in some cases:

ssh server "tail -f /var/log/server.log | grep test"

runs the whole command in the quotes on the server - thus grep waits to fill its buffer.

ssh server tail -f /var/log/server.log | grep test

runs grep on your local machine on the output tail sent through the ssh channel.

The key part here is, that grep adjusts its behaviour depending on whether its stdin is a terminal or not. When you run ssh -t, the remote command is running with a controlling terminal and thus the remote grep behaves like your local one.

Solution 2

check this out: multitail

MultiTail allows you to monitor logfiles and command output in multiple windows in a terminal, colorize, filter and merge.

To tail logs in multiple servers use:

multitail -l 'ssh user@host1 "tail -f /path/to/log/file"' -l 'ssh user@host2 "tail -f /path/to/log/file"'

Solution 3

You can checkout in'side log.

A Java tool I created, able to read local and distant log files using SSH. It is fairly simple to use.

Some more explanations: https://github.com/pschweitz/insidelog/wiki

Just download version corresponding to your operating system, of native jar release executable within your Java Runtime (requires java 8_40 or higher):

https://github.com/pschweitz/insidelog/releases

You can find a complete documentation (embedded with and in Github's page as well)

Comments

[deephacks]

Thanks a lot for the detailed explanation. It make sense to me now and the script works as expected with --line-buffered.

[peterph]

@deephacks In that case please consider accepting the answer - it gives a clue to others having the same problem.

[dave_thompson_085]

grep/glibc's buffering depends on its stdout. ssh tail | grep outputs to the local terminal, unbuffered. ssh -t "tail|grep" outputs to a pty, unbuffered. ssh "tail|grep" outputs to a pipe (to sshd), buffered (unless --line-buffered).

[bishop]

But it doesn't let you do it over ssh, which is a condition of this question. (And, also, the question specifically says "don't want to use multitail".)

[Chris Adams]

@bishop: I think this criticism is unfair in part because while the question may have specified not using multitail, it seems to be due to a misunderstanding. The example above shows how to use arbitrary commands and the regular shell expansions also work — multitail <(ssh …) <(ssh …) — allowing the desired outcome even if it's not how they originally thought the question could be answered.