What's the easiest way to detect what signals are being sent to a process?

Question

I'm trying to debug an issue with my shell (specifically, fish) sending signals to my background processes. I'd like to be able to recognize what signals a process is receiving.

Ideally, I'd like some program that would do something like this:

$ log_signals > signals.txt &

Which would then write the signals it received to its output. However, I don't believe any such program readily exists.

What's the simplest way to trap incoming signals and inspect them, preferably without needing to write my own program?

(I'm running OS X, not Linux, so I'd prefer a more platform-agnostic answer if possible.)

Solution 1

Since Stéphane posted this information as a comment, but I found this information to be more useful than any existing answers, I'm reposting the comment as an answer.

For Linux only: strace prints signals by default, so you can use the -e flag to silence all system calls so that signals are clearer:

strace -e 'trace=!all' cmd

In my case, I was trying to figure out what process was killing a daemon I was running, so I got output that looks like this:

[hendrenj@underling02 ~]$ strace -p 171869 -e 'trace=!all'
Process 171869 attached
--- SIGTERM {si_signo=SIGTERM, si_code=SI_USER, si_pid=151513, si_uid=1000} ---
+++ killed by SIGTERM +++

Here you can see that I attached to PID 171869, and that process was sent SIGTERM by the process with PID 151513.

For further reference, you can check out this blog post, which suggests tools such as auditd (which I believe comes with SELinux if you are on a distro that uses it) or stap (System Tap) if strace isn't powerful enough.

Solution 2

You can use gdb to do this:

gdb --batch -ex 'handle all print' -ex 'handle all nostop' -ex 'handle all pass' -ex 'run' cat will run cat under GDB and print all signals, pass them to the program and not stop execution.

Doing this for a background program is harder, as GDB will try and go into the background instead. You might be better off starting it with cat &, and then attaching the debugger in another window: gdb -ex 'handle all print' -ex 'handle all nostop' -ex 'handle all pass' -ex 'c' (pgrep cat)[1]. Push q[ENTER] three times to get past all the spaff, and then you can watch cat run.

Solution 3

Rather than have some dependency, I realised you can just hook into the trap command to detect any signals. The script below sets up some "traps" on signals (which execute a function when the relevant signal comes in) which then outputs the signal as an exit code. You can change the exit to a console output if you wish.

#!/bin/sh

trap cleanup_1 1 # SIGHUP
trap cleanup_2 2 # SIGINT
trap cleanup_3 3 # SIGQUIT
trap cleanup_6 6 # SIGABRT
trap cleanup_15 15 # SIGTERM

cleanup_1()
{
  exit 1
}

cleanup_2()
{
  exit 2
}

cleanup_3()
{
  exit 3
}

cleanup_6()
{
  exit 6
}

cleanup_15()
{
  exit 15
}

echo "Started"
cat

Solution 4

Use killsnoop

killsnoop traces the kill() syscall, to show signals sent via this method. This may be useful to troubleshoot failing applications, where an unknown mechanism is sending signals.

View more solutions

Comments

[Kusalananda]

Brevity is acceptable, but fuller explanations are better.

[Stéphane Chazelas]

That's Linux-specific though. signal=all is the default. What you want here is trace only signals and not system calls, so it should rather be: strace -ffo logfile -e 'trace=!all' cmdline (still Linux-specific)

[Nils]

To clarify, it's either -p PID or -f CMD, providing command without -f flag doesn't seem to attach strace to the running command.

[jayhendren]

@DmitryVerkhoturov providing a command without -f flag executes the command (i.e. starts a new process) with strace attached. On the other hand, -f and -p are for attaching to processes that are already running.

[Luke Hutchison]

The "this blog post" link is broken.