Keeping a linux process running after I logout

Question

I'm connecting to a Linux machine through SSH, and I'm trying to run a heavy bash script that makes filesystem operations. It's expected to keep running for hours, but I cannot leave the SSH session open because of internet connections issues I have.

I doubt that running the script with the background operator, the ampersand (&), will do the trick, because I tried it and later found that process was not completed. How can I logout and keep the process running?

Solution 1

The best method is to start the process in a terminal multiplexer. Alternatively you can make the process not receive the HUP signal.

---

A terminal multiplexer provides "virtual" terminals which run independent from the "real" terminal (actually all terminals today are "virtual" but that is another topic for another day). The virtual terminal will keep running even if your real terminal is closed with your ssh session.

All processes started from the virtual terminal will keep running with that virtual terminal. When you reconnect to the server you can reconnect to the virtual terminal and everything will be as if nothing happened, other than the time which passed.

Two popular terminal multiplexers are screen and tmux.

Screen has a steep learning curve. Here is a good tutorial with diagrams explaining the concept: http://www.ibm.com/developerworks/aix/library/au-gnu_screen/

---

The HUP signal (or SIGHUP) is sent by the terminal to all its child processes when the terminal is closed. The common action upon receiving SIGHUP is to terminate. Thus when your ssh session gets disconnected all your processes will terminate. To avoid this you can make your processes not receive SIGHUP.

Two easy methods to do so are nohup and disown.

For more information about how nohup and disown works read this question and answer: https://unix.stackexchange.com/questions/3886/difference-between-nohup-disown-and

Note: although the processes will keep running you can no longer interact with them because they are no longer attached to any terminal. This method is mainly useful for long running batch processes which, once started, no longer need any user input.

Solution 2

There are a few ways to do this, but the one I find most useful is to use GNU Screen.

After you ssh in, run screen. This will start another shell running within screen. Run your command, then do a Ctrl-a d.

This will "disconnect" you from the screen session. At this point, you can log out or do anything else you'd like.

When you want to re-connect to the screen session, just run screen -RD from the shell prompt (as the same use user that created the session).

Solution 3

In bash, the disown keyword is perfectly suited to this. First, run your process in the background (either use &, or ^Z then type bg):

$ wget --quiet http://server/some_big_file.zip &
[1] 1156

By typing jobs you can see that the process is still owned by the shell:

$ jobs
[1]+  Running  wget

If you were to log out at this point, the background task would also get killed. However, if you run disown, bash detaches the job and allows it to continue running:

$ disown

You can confirm this:

$ jobs
$ logout

You can even combine the & and disown on the same line, like:

$ wget --quiet http://server/some_big_file.zip & disown
$ logout

This is better than running nohup in my opinion because it doesn't leave nohup.out files littered all over your filesystem. Also, nohup must be run before you run the command — disown can be used if you only decide later on that you want to background and detach the task.

Solution 4

The tool nohup, available on most Linux boxes will do this.

Solution 5

Just to be thorough, I'll point out tmux, which has the same basic idea as screen:

tmux is intended to be a modern, BSD-licensed alternative to programs such as GNU screen. Major features include:

• A powerful, consistent, well-documented and easily scriptable command interface.
• A window may be split horizontally and vertically into panes.
• Panes can be freely moved and resized, or arranged into preset layouts.
• Support for UTF-8 and 256-colour terminals.
• Copy and paste with multiple buffers.
• Interactive menus to select windows, sessions or clients.
• Change the current window by searching for text in the target.
• Terminal locking, manually or after a timeout.
• A clean, easily extended, BSD-licensed codebase, under active development.

It is, however, approximately infinitely easier to search for on Google.

View more solutions

Comments

[gnur]

Using "gnu screen" as your search query works quite well.

[amd]

This is by far the simplest answer. Any output from is automatically directed to nohup.out and can be examined later.

[Mark Booth]

I like this answer as it provides a solution for both interactive and non-interactive situations. In the interactive case, screen gives you a lot more options, but if you are using authorized_keys to allow people to run a script remotely via ssh, the nohup option is a nice simple way for the script to start processes which last longer than the ssh session used to start them.

[Aaron Brown]

nohup does not require zsh at all.

[olafure]

Screen has a whole bunch of command, all starting with Ctrl-a. If you only learn one extra, start with "Ctrl-a ?". Then you'll wan't to learn "Ctrl-a c" and "Ctrl-a n"

[mbq]

+1000 for tmux!

[doc_id]

@olafure +1, thanks. Looks like Screen is going to be of my primary toolbox.

[doc_id]

lesmana, and @Mark Booth I like your answers as it tells more useful tips.. 1+

[fluffy]

Thank you. I came here just to post about dtach too. It's my go-to thing for terminal detachment now; screen does WAY too much, and interferes with input to a ridiculous degree. The fact screen needs its own termcap is pretty unsettling.

[Zenon]

nohup is found on far more machines then screen, so you should know how to use it.

[Mark Booth]

@rahmanisback - Remember that you can change your accepted answer at any time. Just because so far EricA's answer is voted most highly doesn't mean it's the best answer for you, indeed making it your accepted answer may well encourage more people to vote it up as a good answer.

[EEAA]

Well, I lost an accepted answer, but gained the gold "populist" badge. :) w00t.

[crasic]

*cough*tmuxisbetter*cough*

[Jeremy Visser]

Yes – this is bash–specific, as bash is the only shell I've ever used. I wonder if other shells support anything similar (i.e. launching in background without nohup) – it would be fantastic if someone could post other answers for other shells.

[h0tw1r3]

tmux > screen. Try it, you'll never go back.

[h0tw1r3]

tmux > screen. Try it, you'll never go back.

[w00t]

see my answer showing how to do it it with any sh-lookalike

[Bryan Mills]

byobu > screen. Tried it, never turned back

[EEAA]

@TheLQ - byobu is GNU Screen. You're still using screen, just with a highly-customized .screenrc.

[doc_id]

Interesting. That emerged some notable questions. I can see that you set the STDIN to nothing, the - operator? What is the difference between < /dev/null and &- ? I guess that STDIN (and others STDOUT and STDERR) could be assigned either a file by < file, or a stream by <& stream in case of STDIN. Would it be the same using < /dev/null in your example above? And Does the operator - above refer to a null as the stream?

[w00t]

When you do x<&-, that closes file descriptor x. In this case there is no x, which makes bash default to 1, i.e. standard input. If you use </dev/null you're not closing stdin, you're just giving an empty file to the program as input.

[w00t]

And to be honest I don't really know why this daemonizes the thing you're running :-) It does work though, we use it in production. I discovered it while messing around hoping I could daemonize a process in shell without needing anything special - so I started by closing stdin and that was enough. I should go read some shell sources but I presume that if you close stdin and background the process, it also detaches the process.

[Jeremy Visser]

I think the reason is that a SIGHUP (the actual signal that causes the child to quit when the shell dies) is triggered when a parent process closes its child's stdin handle. However, if stdin starts out as null, rather than being closed after the fact, there is no way for the parent to trigger the SIGHUP. Nice find, though — never would have thought of that.

[w00t]

@JeremyVisser that sounds really plausible!

[Bryan Hunt]

+1 for tmux. I gave up screen 5 weeks ago.

[code_monk]

+1 because of being able to decide later. Just this moment i had a need for this. worked as advertised

[Cees Timmerman]

screen is dead. tmux is king. Old hats prefer nohup.

[chicks]

bash, ksh, csh, etc. are all job control shells. None of these require nohup for background jobs to continue running. If you are stuck with pre-Bourne UNIX shells then keep using nohup.

[chicks]

nohup is not needed in modern shells.

[code_monk]

tmux FTW ☺︎ ☺︎☺︎

[user45793]

The dtach link is dead.

[user45793]

Perhaps now at github.com/crigler/dtach ?