Can I nohup/screen an already-started process?

Question

I'm doing some test-runs of long-running data migration scripts, over SSH. Let's say I start running a script around 4 PM; now, 6 PM rolls around, and I'm cursing myself for not doing this all in screen.

Is there any way to "retroactively" nohup a process, or do I need to leave my computer online all night? If it's not possible to attach screen to/nohup a process I've already started, then why? Something to do with how parent/child proceses interact? (I won't accept a "no" answer that doesn't at least address the question of 'why' -- sorry ;) )

Solution 1

If you're using Bash, you can run disown -h job

disown

disown [-ar] [-h] [jobspec ...]

Without options, each jobspec is removed from the table of active jobs. If the -h option is given, the job is not removed from the table, but is marked so that SIGHUP is not sent to the job if the shell receives a SIGHUP. If jobspec is not present, and neither the -a nor -r option is supplied, the current job is used. If no jobspec is supplied, the -a option means to remove or mark all jobs; the -r option without a jobspec argument restricts operation to running jobs.

Solution 2

Use reptyr

From the README:

reptyr - A tool for "re-ptying" programs.
-----------------------------------------

reptyr is a utility for taking an existing running program and
attaching it to a new terminal. Started a long-running process over
ssh, but have to leave and don't want to interrupt it? Just start a
screen, use reptyr to grab it, and then kill the ssh session and head
on home.

USAGE
-----

  reptyr PID

"reptyr PID" will grab the process with id PID and attach it to your
current terminal.

After attaching, the process will take input from and write output to
the new terminal, including ^C and ^Z. (Unfortunately, if you
background it, you will still have to run "bg" or "fg" in the old
terminal. This is likely impossible to fix in a reasonable way without
patching your shell.)

A few blog posts by its author:

• reptyr: Attach a running process to a new terminal
• reptyr: Changing a process’s controlling terminal

Solution 3

To steal a process from one tty to your current tty, you may want to try this hack:

http://www.ucc.asn.au/~dagobah/things/grab.c

It needs some reformatting in order to compile to current Linux/glibc versions, but still works.

Solution 4

When a process starts, STDIN, STDOUT and STDERR are connected to something. Generally you can't change that once the command is started. In the case you're describing, that's probably a tty associated with the ssh session. nohup pretty much just does ...

command < /dev/null > nohup.out 2>&1

That is, sets STDIN to /dev/null, STDOUT to a file and STDERR to STDOUT. Screen does much more sophisticated things involving setting up ttys that direct to itself.

I don't know of any way to retroactively nohup or screenize a running process. If you cd to /proc/$pid/fd and see what 0, 1 and 2 point to.

You might have some luck with disown, but not if the process tries to do anything with STDIN, STDOUT or STDERR.

Solution 5

Cryopid is a further development from the author of grab.c that freezes a process to a file, which you then run (inside screen) to resume the process.

View more solutions

Comments

[ncoder]

Just saw an interesting blog post about disown. blogs.oracle.com/ksplice/entry/disown_zombie_children_and_th‌​e

[Toby]

That would still kill the process when the owning tty dies so the OP needs to leave the box where he intiated the command running, which is what he wants to avoid

[Chris_K]

OK I'll buy that. Thanks for the explanation. So the disown -h suggestions sure look a lot smarter than mine :-)

[ncoder]

It's the logout thing.

[ncoder]

Awesome; I was hoping something like this would turn up.

[ncoder]

+1 for the good comments. st(din|out|err) is the other half of the problem, and I appreciate the advice on where to start looking, next time I'm in this jam.

[Toby]

Life can be unfair. gharper and me were posting this at about the same time :)

[ncoder]

Exceptionally cool.

[ncoder]

Sorry for revoking the "answer" cred, but damn... a good one came up. I don't feel too bad, though; you're already at +12 from this ;)

[gharper]

Totally agree - I'm going to have to play with that grab.c program a bit. Looks very cool!

[Zaid Amir]

Nice! I tried to use cryopid in my master's dissertation about process migration, but it failed to work all the times, no matter what I did. In the end, I had to use dynckpt with an ancient version of Linux. Are you perhaps involved in the development of cryopid? I see that your name is similar to the author's domain.

[feverzsj]

I'm not involved in the development, I just know the author from university. He doesn't have the time to maintain cryopid at the moment, so it looks like some people started working on it at sharesource.org/project/cryopid

[Zan Lynx]

You actually can change it on most Unixes. It's a disgusting hack. I love it. :) What you do is, connect to the process using debug support like ptrace, then force the process to call dup2() to reconnect 0,1,2 to another filehandle.

[Araejay]

Another way to stop it disconnecting is to run top. THat's always shuffeling a few bytes around.

[Thomas Dignan]

You are my hero

[Phil P]

disown is not specific to bash. It's also in zsh, ksh93, ...

[ncoder]

I'm going to stick with built-in tools (i.e. disown), but it's not as flexible as reptyr. +1

[Andrew Core]

You could still do this, just after you'd have to disown -h: stackoverflow.com/a/625436/705198

[Matthew Setter]

thanks @gharper it works like a treat. I've got a process that was already running i estimate will take over 5 hours to complete and I can't be here to babysit it.

[Shoan]

Is the jobspec the same as the process pid?

[mltsy]

I found that you actually have to use disown %1 if 1 is the jobspec, unlike fg or bg where you just use bg 1 serverwatch.com/tutorials/article.php/3935306/…

[Rich]

I tried this, but the program will die the first time it writes to stdout. You need to use reptyr - see Jonathan Tran's answer

[Michael Martinez]

yeah, you can change it. Involves pausing the process (SIGSTOP), and modifying the file descriptors for fd 0, 1, 2. Then restarting (SIGCONT).

[Cees Timmerman]

@Rich, so stackoverflow.com/a/625436/819417 won't work?

[Rich]

@CeesTimmerman no, in the case where you start a process (without using screen) and then want to log out of SSH, disown won't cut it, and reptyr is needed. I tested that on RHEL in 2013; perhaps things are different now (seems unlikely though).

[ncoder]

tmux is great -- but, like nohup or screen, it only works if you use it before starting your process. This question is about the times when you realize you need tmux after your process is already running.

[Joachim Wagner]

github.com/nelhage/reptyr/issues/54 (Maybe -T would have helped. Only learned about it after failed attempt.)