How to terminate remotely called "tail -f" when connection is closed?

Question

I just noticed that if I execute ssh user@remote_host tail -f /some/file , then tail -f /some/file keeps running on the remote_host even if ssh connection is closed!

So, after several connects and disconnects, number of running tail -f /some/file grows. How to actually terminate tail -f when ssh connection is closed?

Solution 1

In

ssh host tail -f file

The ssh client connects to the sshd server on host over a TCP connection. sshd runs tail -f with its stdout redirected to a pipe. sshd reads what's coming from the other end of the pipe and encapsulates it in the sshd protocol to send to the ssh client. (with rshd, tail stdout would have been the socket directly, but sshd adds encryption and is able to multiplex several streams (like for port/agent/X11/tunnel redirection, stderr) on a single TCP connection so has to resort to pipes).

When you press CTRL-C, a SIGINT is sent to the ssh client. That causes ssh to die. Upon dying the TCP connection is closed. And therefore, on host, sshd dies as well. tail is not killed, but its stdout is now a pipe with no reader at the other end. So, the next time it writes something to its stdout, it will receive a SIGPIPE and die.

In:

ssh -t host 'tail -f file'

It's the same thing except that instead of being with a pipe, the communication between sshd and tail is via a pseudo-terminal. tail's stdout is a slave pseudo-terminal (like /dev/pts/12) and whatever tail write there is read on the master side (possibly modified by the tty line discipline) by sshd and sent encapsulated to the ssh client.

On the client side, with -t, ssh puts the terminal in raw mode. In particular, that disables the terminal canonical mode and terminal signal handling.

So, when you press Ctrl+C, instead of the client's terminal line discipline sending a SIGINT to the ssh job, that just sends the ^C character over the connection to sshd and sshd writes that ^C to the master side of the remote terminal. And the line discipline of the remote terminal sends a SIGINT to tail. tail then dies, and sshd exits and closes the connection and ssh terminates (if it's not otherwise still busy with port forwardings or other).

Also, with -t, if the ssh client dies (for instance if you enter ~.), the connection is closed and sshd dies. As a result, a SIGHUP will be sent to tail.

Now, beware that using -t has side effects. For instance, with the default terminal settings, \n characters are converted to \r\n and more things may happen depending on the remote system, so you may want to issue a stty -opost (to disable output post-processing) on the remote host if that output is not intended for a terminal:

$ ssh  localhost 'echo x' | hd
00000000  78 0a                                             |x.|
00000002
$ ssh -t localhost 'echo x' | hd
00000000  78 0d 0a                                          |x..|
00000003
$ ssh -t localhost 'stty -opost; echo x' | hd
00000000  78 0a                                             |x.|
00000002

Another drawback of using -t/-tt is that stdout and stderr are not differentiated on the client. Both the stdout and stderr of the remote command will be written to the ssh client's stdout:

$ ssh localhost ls /x  | wc -l
ls: cannot access /x: No such file or directory
0
$ ssh -t localhost ls /x | wc -l
1

Solution 2

You need terminal allocation on the remote side:

ssh -t user@remote_host tail -f /some/file

or even

ssh -tt user@remote_host tail -f /some/file

Comments

[Dmitry Frank]

Thanks, both -t or -tt works. But I still can't understand real reason of this: say, when I call shell remotely and close the connection, shell is terminated. But tail -f is not. Of course I have already read about -t option in man ssh, but it didn't help much. It seems I don't understand some generics, and I'd be happy if you suggest some docs to read about it, or probably explain it yourself. Thanks!

[Hauke Laging]

@DmitryFrank My understanding is this: If the connection breaks then sshd sends a SIGHUP. But where no terminal is there can be no terminal connection hangup...

[Dmitry Frank]

Thanks, I will read about SIGHUP and other signals, still know almost nothing about it.

[Dmitry Frank]

fyi: I just tried to run tail -f, then I've opened htop and sent SIGHUP to it (by pressing F9 -> 1 -> Enter), and tail -f is terminated! So, reason should be some different..

[Hauke Laging]

@DmitryFrank You have misunderstood the problem. The problem is not that tail would not react to SIGHUP. The problem is that SIGHUP is **not sent` to tail without a pseudo terminal. You can see that by attaching strace to tail in both cases.

[Dmitry Frank]

Thank you very much for such a detailed explanation! I wish I could accept two answers..

[x-yuri]

"with -t, if the ssh client dies (for instance if you enter ~.)" What is ~. again?

[Stéphane Chazelas]

@x-yuri, ~. is the escape sequence you enter to disconnect the client. See man ssh for details.