Prevent currently running linux task to be killed after logout from SSH

Question

I have a computational task running for a few days on a linux server via SSH connection. It is not on the background, so the SSH connection is under hold of the task.

I want to restart the local machine (not the server) from which I have opened ssh session, but want to keep the task running. Is that any possible?

Solution 1

If your task is already launched, it is too late* to consider alternative solutions that insert an additional layer between your ssh session and the shell running the command, like screen, tmux, byobu, nohup and the likes.

If your process support to be placed in the background and particularly doesn't hang when stdout and stderr are unwritable/closed, you can put it in the background before logging out with ControlZ and bg then detach it from your shell with the disown builtin.

eg:

$ ssh localhost
You have new mail.
Last login: Fri Jun  6 11:26:56 2014

$ /bin/sleep 3600    


^Z[1] + Stopped                  /bin/sleep 3600
$ bg
[1] /bin/sleep 3600&
$ jobs
[1] +  Running                 /bin/sleep 3600
$ disown %1
$ exit
Connection to localhost closed.
$ ps -ef|grep sleep
jlliagre 12864     1  0 21:12 ?        00:00:00 /bin/sleep 3600
jlliagre 13056 12477  0 21:13 pts/18   00:00:00 grep sleep
$ 

* As Bob commented, there are actually several hackish ways to reparent a tty session under Linux. repty, retty, injcode and neercs. The most advanced looks to be reptyr but you might need root privileges to enable ptrace to hack your process.

Solution 2

One solution is to use GNU screen. You could start up screen, run your command, then detach with C-a d. Later, to reconnect, do screen -r, and you are back in your previous session.

Other benefits of screen are window management (so you can switch to other shells while your command is running, without needing a new SSH connection), and it allows your command to remain in the foreground, whether in the current session or a later one.

Edit: As noted in the comments, this will only work if you remember to start screen before running the command. If the command is already running, then you will need @jlliagre's solution.

Solution 3

One of the "standard" ways to do so, is to use the nohup command, included in coreutils, like this:

nohup COMMAND [ARGS] &

But the command will redirect the output (STDOUT & STDERR AFAIK) of the program into a file nohup.out, making it somehow annoying sometimes (like generating a huge log file), so you may want to make your own redirection, or redirect it to /dev/null if you want.

Comments

[Ali]

I used Ctrl+Z, the job stopped and moved to background, then performed disown command. It returned: bash: warning: deleting stopped job 1 with process group 24876. Now my job is listed in ps -all but seem not working (CPU usage is 0%)

[user1984103]

tmux is another program like screen that allows you to have a remote session which is not tied to your current SSH connection.

[Ali]

Seems my process is stopped. I guess I should have provided proces ID with disown command

[Scott Weldon]

Ah, okay. In that case, I think you will need to use @jlliagre's solution. (As far as I know, there is not a way to connect an already running process to something like screen or tmux.)

[wangguoqin1001]

/ashamed didn't read carefully. Sorry.

[Jason Zhu]

@Ali I think you forgot to run the bg command to resume the suspended job.

[jlliagre]

@JasonZhu Thanks for pointing that. It was unclear the bg command was required in the first part of my reply. Edited.

[tvdo]

> it is too late to consider alternative solutions that insert an additional layer between your ssh session and the shell running the command -- not really. There's tools like reptyr and retty that can be used to do that.

[kasperd]

If you forgot the bg command and only ran the disown command, you should be able to get the process running again with kill -CONT which does roughly the same that bg would.

[mike]

Actually nohup can be used for this. Just suspend the running process and send it to background (Ctrl+ Z and run bg) and then you can issue nohup %1.