Get ssh to forward signals

Question

I want to be able to send signals (SIGINT is the most important) through ssh.

This command:

ssh server "sleep 1000;echo f" > foo

will start sleep on server and after 1000 seconds it will put 'f\n' in the file foo on my local machine. If I press CTRL-C (i.e. send SIGINT to ssh) it will kill ssh, but it will not kill sleep on the remote server. I want it to kill sleep on the remote server.

So I tried:

ssh server -t "sleep 1000;echo f" > foo

But if stdin is not a terminal I get this error:

Pseudo-terminal will not be allocated because stdin is not a terminal.

and then SIGINT is still not forwarded.

So I tried:

ssh server -t -t "sleep 1000;echo f" > output

But then the output in foo is not 'f\n' but instead 'f\r\n' which is disastrous in my situation (as my output is binary data).

In the above I use "sleep 1000;echo f", but in reality that is supplied by the user, thus it can contain anything. However, if we can make it work for "sleep 1000;echo f" we can most likely make it work for all realistic situations.

I really do not care about getting a pseudo-terminal at the other end, but I have been unable to find any other way of getting ssh to forward my SIGINT.

Is there another way?

Edit:

The user could give commands that read binary data from stdin, such as:

seq 1000 | gzip | ssh server "zcat|bzip2; sleep 1000" | bzcat > foo

The user could give commands that are cpu intensive, such as:

ssh server "timeout 1000 burnP6"

Edit2:

The version that seems to work for me is:

your_preprocessing |
  uuencode a | ssh -tt -oLogLevel=quiet server "stty isig -echoctl -echo ; uudecode -o - |
your_command |
  uuencode a" | uudecode -o - |
your_postprocessing

Thanks to digital_infinity for pointing me in the right direction.

Solution 1

Short answer:

ssh -t fs "stty isig intr ^N -echoctl ; trap '/bin/true' SIGINT; sleep 1000; echo f" > foo

and stop the program by CTRL+N.

Long explanation:

1. You must use stty option intr to change your server or local interrupt character to not collide with each other. In the command above I've changed the server interrupt character to CTRL+N. You can change your local interrupt character and leave the server's one without any changes.
2. If you don't want the interrupt character to be in your output (and any other control character) use stty -echoctl.
3. You must assure that control characters are switched on on the server bash invoked by sshd . If you don't you can end up with processes still hanging around after you logout. stty isig
4. You actually catch SIGINT signal by trap '/bin/true' SIGINT with empty statement. Without the trap you will not have any stdout after SIGINT signal on your end.

Solution 2

The solution evolved into https://www.gnu.org/software/parallel/parallel_design.html#The-remote-system-wrapper

$SIG{CHLD} = sub { $done = 1; };
$pid = fork;
unless($pid) {
    # Make own process group to be able to kill HUP it later
    setpgrp;
    exec $ENV{SHELL}, "-c", ($bashfunc."@ARGV");
    die "exec: $!\n";
}
do {
    # Parent is not init (ppid=1), so sshd is alive
    # Exponential sleep up to 1 sec
    $s = $s < 1 ? 0.001 + $s * 1.03 : $s;
    select(undef, undef, undef, $s);
} until ($done || getppid == 1);
# Kill HUP the process group if job not done
kill(SIGHUP, -${pid}) unless $done;
wait;
exit ($?&127 ? 128+($?&127) : 1+$?>>8)

Solution 3

I tried all the solutions and this was the best:

ssh host "sleep 99 < <(cat; kill -INT 0)" <&1

https://stackoverflow.com/questions/3235180/starting-a-process-over-ssh-using-bash-and-then-killing-it-on-sigint/25882610#25882610

Solution 4

I think you could find PID of the process you're running on server and send a signal using another ssh command (like this: ssh server "kill -2 PID").

I use this method for sending reconfiguration signals to applications running on a different machine (my applications catch SIGUSR1 and read a config file). In my case finding PID is easy, because I have unique process names and I can find the PID by sending a ps request via ssh.

View more solutions

Comments

[Warren Young]

I think your actual usage of ssh must be more complicated than what you show as examples, because you can get the behavior you want with a simple rearrangement: sleep 1000 && ssh server "echo f" > foo (It has to be &&, not ;, so that killing sleep prevents the ssh command from running.) If I'm right, please make your examples more representative of your actual usage, so a better answer can be given.

[Ole Tange]

Correct: sleep and echo are actually user supplied scripts and not literally sleep and echo. So we do not know what they do and should assume the worst.

[Warren Young]

Sooo...you're going to come up with a better example command, right? One where a simple rearrangement doesn't fix the problem? You ask a good question, and I'd like to see it answered, but you're less likely to get an answer if "so don't do that, then" is a reasonable reply.

[Ole Tange]

As elabrated in the question: """In the above I use "sleep 1000;echo f", but in reality that is supplied by the user, thus it can contain anything"""

[Ole Tange]

If you insist on a real world example, use: eval echo $SHELL | grep -E "/(t)?csh" > /dev/null && echo setenv PARALLEL_SEQ '$PARALLEL_SEQ'\; setenv PARALLEL_PID '$PARALLEL_PID' || echo PARALLEL_SEQ='$PARALLEL_SEQ'\;export PARALLEL_SEQ\;PARALLEL_PID='$PARALLEL_PID'\;export PARALLEL_PID ;' zcat\ \$PARALLEL_SEQ\ log.gz\ \|\ bzip2

[Warren Young]

Why can't you write out a temporary script, scp it to the remote, and execute it in place? Killing the client side ssh will stop the server-side script from producing output then, wouldn't it?

[Ole Tange]

The problem is not stopping the remote process from producing output. The problem is stopping the process. It could very well be a very cpu intensive task that will run for hours without reading or writing.

[Ole Tange]

It seems not to deal nicely with binary input: seq 1000 | gzip | ssh -t -t server "stty isig intr ^N -echoctl ; trap '/bin/true' SIGINT; sleep 1; zcat|bzip2" | bzcat > foo;

[Ole Tange]

I do not see a bullet proof way of finding the PID of the program being run on remote. Remember it is given by the user. It could be: ssh server 'exec $(echo fyrrc 1000| /usr/games/rot13)'

[monkeyhouse]

You cannot interrupt by console if you set standard input for ssh for something other then console. In this case you must interrupt by the stdin stream.

[monkeyhouse]

I found that this seq 1000 | gzip | ssh -tt dell-test "zcat|bzip2" | bzcat > foo does not work too. To have this command work we need to remove -tt . So the pseudo terminal allocation probably takes some input from stdin

[Ole Tange]

I tried the uuencode. In this version it does not work: cat foo.gz | perl -ne 'print pack("u",$_)' | ssh -t -t server 'perl -ne "print unpack(\"u\",\$_)"| zcat | gzip | perl -ne "print pack(\"u\",\$_)"' | perl -ne 'print unpack("u",$_)' > foo2.gz

[monkeyhouse]

I have this: (sleep 1; seq 1000 ) | gzip | uuencode bin | ssh -tt dell-test "stty isig intr ^N -echoctl -echo ; trap '/bin/true' SIGINT; sleep 1; uudecode -o /dev/stdout | zcat |bzip2 | uuencode bin2 " | uudecode -o /dev/stdout | bzcat >foo . Although the interrupt character does not work - we need some transmission method for interrupt character while stdin is busy.

[Ole Tange]

This seems to work: (sleep 1; seq 1000 ) | gzip | uuencode bin | ssh -tt server "stty isig intr ^C -echoctl -echo ; sleep 1; uudecode -o /dev/stdout | zcat |sort -r|bzip2 | uuencode bin2 " | uudecode -o /dev/stdout | bzcat >foo

[Ole Tange]

I do now quite understand why you need the trap. Can you give a situation where it fails if you do not have the trap?

[Ole Tange]

I believe this is the best so far: (sleep 1; seq 1000 ) | gzip | uuencode bin | ssh -tt -oLogLevel=quiet server "stty isig -echoctl -echo ; uudecode -o /dev/stdout | zcat |sort -r|bzip2 | uuencode bin2 " | uudecode -o /dev/stdout | bzcat >foo

[monkeyhouse]

I need trap because without it the command after interrupted command is not executed after SIGINT (in my answer echo f is never executed if there is no trap). I don't know if you want to achieve this effect. I tested on Linux - OpenSuSE 12.1.

[dhag]

Hi! I think you could make your answer a lot more useful by explaining, in as much detail as you deem relevant, how it works. Don't hesitate to edit it to insert new information.

[blueyed]

That does not stop the sleep process in case the connection is lost.

[Eric Woodruff]

When the connection is lost the stdin is broken unblocking the cat which will kill the process group. Whether that INTerrupt signal is good enough for your task is a different question.

[blueyed]

It should be good enough for the sleep, shouldn't it? I've added some date >> /tmp/killed after the cat, but it was not triggered. Is there some timeout involved? I'm using Zsh normally, but also tested it with bash as the remote login shell.

[Eric Woodruff]

I think you are at the mercy of the connection timeout.

[blueyed]

Are there settings to control this? For the client side -o ServerAliveInterval=3 -o ServerAliveCountMax=2 allows to detect it quickly, but is there something for the server side?

[Eric Woodruff]

There is a ClientAliveInterval too, but those just send null packets to keep the connection open in case something in between is looking to close it (like a proxy). You might want TCPKeepAlive unix.stackexchange.com/a/150406/84827