How to find out what process is writing to STDOUT?

Question

I have two instances of a process running. One of them is "frEAkIng oUT!" and printing errors non stop to STDOUT.

I want to kill the broken process but I have to make sure I don't terminate the wrong one. They were both started about at the same time and using top I can see they both use about the same amount of memory and CPU. I can't seem to find anything that points to which process is behaving badly.

The safest thing would be to figure out which process/pid is writing to STDOUT.

Is there any way to do that?

Solution 1

On Linux, assuming you want to know what is writing to the same resource as your shell's stdout is connected to, you could do:

strace -fe write $(lsof -t "/proc/$$/fd/1" | sed 's/^/-p/')

That would report the write() system calls (on any file descriptor) of every process that have at least one file descriptor open on the same file as fd 1 of your shell.

Solution 2

You can stop both processing by sending them SIGSTOP (replace pid1 and pid2 by the actual PIDs or use killall and the application name):

kill -SIGSTOP pid1 pid2

The printing on the terminal (or wherever stdout is redirected to) should stop. Then continue one of them using

kill -SIGCONT pid1

If the error messages appear immediately, you know its the first process. If not you can stop it again and continue the second...

Before killing a stopped process, it is good practise to send first SIGCONT.

The same technique can be used with Ctrl-Z and the shell job controls (fg %1, bg %1, kill %1, ...).

Comments

[TCZ8]

Thats a very good way to troubleshoot it but i was really looking for a way to trace who is writting to terminal. Thank you

[TCZ8]

That is what I had in mind initially, I will try both methods thanks to you both.

[TCZ8]

Was just rereading this question, seems like I had a brain fart while writing my last comment. This would also fix my problem. Thx.