How can I redirect an already running process's stdout/stderr?

Question

(Running on an Ubuntu 10.04 64-bit server)

Yesterday, I made the mistake of starting a process (which I didn't realize was going to take several days to run) over SSH without using screen. I've spent all day today trying to figure out some way that I can pry the process's output from SSH's iron grasp so that I can reboot my client machine, but am not getting anywhere.

I tried using gdb and following the instructions at the bottom of this page, but when I run the first gdb command to create a file, I get an error message stating No symbol table is loaded. Use the "file" command. From what I gathered, this means I have to recompile the program whose output I'm trying to redirect, which of course is absolutely no help to me now that it's already running.

I also thought I might be able to use retty to redirect output to another terminal, but evidently it does not compile on 64-bit platforms.

How can I redirect this process's output to another terminal or to a file?

Solution

It looks like those instructions given for gdb are incorrect and incomplete in several ways.

First, you need to use

gdb [executablefile] [pid]

so that GDB knows what the program it's connecting to actually is. I just tested this, and it can use a stripped executable. Note that while gdb is attached to the process the process will be suspended. If this process is communicating over a network, type quickly or the network connection will probably timeout.

Second, the commands given don't explain what they're doing, and the instruction that you should cd "to a directory that you want your program to write files to" is wrong, since gdb is asking the original program to execute the creat() function. The example given will create myprog.stderr and myprog.stdout files in the Current Working Directory of the running program, not in the directory you ran gdb. Use absolute pathnames here if you don't know what that program's CWD is (or look at ls -l /proc/[pid]/cwd).

Third, running with the lack of explanation, it's important to know that the first parameter to dup2() is the file descriptor number returned by the previous creat() so if this running program had several files open you might end up with an exchange like

(gdb) call creat("/home/me/myprog.stdout",0600)
$1 = 7
(gdb) call dup2(7,1)
$2 = 1
(gdb) call creat("/home/me/myprog.stderr",0600)
$3 = 8
(gdb) call dup2(8,2)
$4 = 2

When you quit gdb, it'll ask you if you want to "quit anyway (and detach it)" the answer is yes.

Finally, bg and disown are bash builtins. If you weren't using bash, then you're on your own from here. bg moves a suspended job to the background as if it were started there using somecommand &, and disown removes the program from bash's list of active programs to SIGHUP when bash exits.

Comments

[nonoitall]

That's what I've been doing. I still need to know what will work server-side though.

[nonoitall]

I followed your instructions, altering the commands accordingly and the commands succeeded this time! However, nothing is being written to the files that were created. Currently, ls -l /proc/[pid]/fd reads as follows (minus the irrelevant bits): 0 -> /dev/pts/0 (deleted); 1 -> /dev/pts/0 (deleted); 2 -> /dev/pts/0 (deleted); 3 -> /dev/sdb; 4 -> /root/myprog.stdout; 5 -> /root/myprog.stdout; 6 -> /root/myprog.stderr; 7 -> /root/myprog.stderr (And in case you're wondering, the process in question is running as root, so it should be able to write to those files.)

[DerfK]

Something went wrong with call dup2() then, it should have duplicated the fd for "myprog.stdout" over fd 1 rather than making a new fd 5. You should still be able to fix this if you redo gdb and call dup2(4,1) and call dup2(6,2). Hopefully the extra copy of the file being open won't interfere, but don't try call close(5) on the spare copy, I just tested that out and the program immedeately segfaulted.

[nonoitall]

Ah, I see what went wrong. I was in a hurry to enter in the gdb commands and I left out the '2' in 'dup2'. All seems to be right now though as the output is getting written to the stderr file I created. There are still a couple extra descriptors open, but it doesn't seem like they're causing any harm. Thanks!