What does a program do when it's sent SIGKILL signal?

Question

When I used killall -9 name to kill a program, the state become zombie. Some minutes later, it stopped really. So, what's happening during those minutes?

Solution

The program actually never receives the SIGKILL signal, as SIGKILL is completely handled by the operating system/kernel.

When SIGKILL for a specific process is sent, the kernel's scheduler immediately stops giving that process any more CPU time for running user-space code. If the process has any threads executing user-space code on other CPUs/cores at the time the scheduler makes this decision, those threads will be stopped too. (In single-core systems this used to be much simpler: if the only CPU core in the system was running the scheduler, it by definition wasn't running the process at the same time!)

If the process/thread is executing kernel code (e.g. a system call, or an I/O operation associated with a memory-mapped file) at the time of SIGKILL, it gets a bit trickier: only some system calls are interruptible, so the kernel internally marks the process as being in a special "dying" state until the system calls or I/O operations are resolved. CPU time to resolve those will be scheduled as usual. Interruptible system calls or I/O operations will check if the process that called them is dying at any suitable stopping points, and will exit early in that case. Uninterruptible operations will run into completion, and will check for a "dying" state just before returning to user-space code.

Once any in-process kernel routines are resolved, the process state is changed from "dying" to "dead" and the kernel begins cleaning it up, similar to when a program exits normally. Once the clean-up is complete, a greater-than-128 result code will be assigned (to indicate that the process was killed by a signal; see this answer for the messy details), and the process will transition into "zombie" state. The parent of the killed process will be notified with a SIGCHLD signal.

As a result, the process itself will never get the chance to actually process the information that it has received a SIGKILL.

When a process is in a "zombie" state it means the process is already dead, but its parent process has not yet acknowledged this by reading the exit code of the dead process using the wait(2) system call. Basically the only resource a zombie process is consuming any more is a slot in the process table that holds its PID, the exit code and some other "vital statistics" of the process at the time of its death.

If the parent process dies before its children, the orphaned child processes are automatically adopted by PID #1, which has a special duty to keep calling wait(2) so that any orphaned processes won't stick around as zombies.

If it takes several minutes for a zombie process to clear, it suggests that the parent process of the zombie is struggling or not doing its job properly.

There is a tongue-in-cheek description on what to do in case of zombie problems in Unix-like operating systems: "You cannot do anything for the zombies themselves, as they are already dead. Instead, kill the evil zombie master!" (i.e. the parent process of the troublesome zombies)

Comments

[gidds]

What happens if the process is in a kernel call (e.g. doing I/O) when SIGKILL is sent?

[haikun he]

I think this process will become zombie, and his parent process wait the kernel call finished to recycle the resources.(English is poor,please forgive me.)

[zwol]

@gidds Either the I/O will be cancelled in order to execute the SIGKILL, or the SIGKILL will be delayed until the I/O completes. This is the difference between 'S' and 'D' sleep states in ps: 'S' is for I/O waits that the kernel can cancel in order to deliver a signal, and 'D' for those it can't.

[Rob C]

It's not entirely accurate to say the schedule immediately stops giving the process CPU time. The kernel side of the signal handling is still executed by that process, but the process will only be executing kernel code so you are right when you say the program never receives the signal. The process will be executing kernel code responsible for most of the cleanup of resources (open files, virtual memory, etc.) The last steps of this cleanup code is to change the process state to zombie and invoke the scheduler. Then it will never be scheduled again.

[Rob C]

@gidds There are at least four different states that process can be in. It can be running kernel code at the moment or it can be sleeping in one of three different sleep states. The sleep states can either be interruptible, non-interruptible, or non-interruptible except for deadly signals. If it is in non-interruptible sleep it will be left sleeping for as long as it needs and only once it wakes up will it have a chance to die. If it was in one of the other two sleep states it will be woken up immediately and scheduled as soon as there is a CPU available for it.

[Rob C]

@gidds What happens next depends on the kernel code it was running. Regardless of whether it was already running or first had to be woken up and then could start running the kernel code it was in at the time will be allowed to continue. And that kernel code is responsible for noticing that the process has been told to die and act accordingly. Most of the time the proper way to deal with that in kernel code is to just return an error from whatever function it was executing. Once the kernel call stack has been unwound the signal handling code can take over just before returning to user mode.

[David Conrad]

Typo: "its currently executing threads, if there are any"

[MrSpreadsheet]

I didnt realize I was so interested in this topic. Great answer!