When does the system send a SIGTERM to a process?

Question

My server program received a SIGTERM and stopped (with exit code 0). I am surprised by this, as I am pretty sure that there was plenty of memory for it. Under what conditions does linux (busybox) send a SIGTERM to a process?

Solution 1

_{I'll post this as an answer so that there's some kind of resolution if this turns out to be the issue.}

An exit status of 0 means a normal exit from a successful program. An exiting program can choose any integer between 0 and 255 as its exit status. Conventionally, programs use small values. Values 126 and above are used by the shell to report special conditions, so it's best to avoid them.

At the C API level, programs report a 16-bit status¹ that encodes both the program's exit status and the signal that killed it, if any.

In the shell, a command's exit status (saved in $?) conflates the actual exit status of the program and the signal value: if a program is killed by a signal, $? is set to a value greater than 128 (with most shells, this value is 128 plus the signal number; ATT ksh uses 256 + signal number and yash uses 384 + signal number, which avoids the ambiguity, but the other shells haven't followed suit).

In particular, if $? is 0, your program exited normally.

Note that this includes the case of a process that receives SIGTERM, but has a signal handler for it, and eventually exits normally (perhaps as an indirect consequence of the SIGTERM signal, perhaps not).

---

To answer the question in your title, SIGTERM is never sent automatically by the system. There are a few signals that are sent automatically like SIGHUP when a terminal goes away, SIGSEGV/SIGBUS/SIGILL when a process does things it shouldn't be doing, SIGPIPE when it writes to a broken pipe/socket, etc. And there are a few signals that are sent due to a key press in a terminal, mainly SIGINT for Ctrl+C, SIGQUIT for Ctrl+\ and SIGTSTP for Ctrl+Z, but SIGTERM is not one of those. If a process receives SIGTERM, some other process sent that signal.

¹ _{roughly speaking}

Solution 2

SIGTERM is the signal that is typically used to administratively terminate a process.

That's not a signal that the kernel would send, but that's the signal a process would typically send to terminate (gracefully) another process.

That's the signal that is sent by default by the kill, pkill, killall... commands.

That's the signal that is sent to daemons to stop them (like upon a service some-service stop), or sent by init before shutdown (followed by SIGKILL for those processes that have not managed to terminate in time upon SIGTERM).

Note that SIGTERM is not the signal that is sent upon ^C. The signal sent upon ^C is SIGINT.

Comments

[Gilles 'SO- stop being evil']

I can't think of any case when the kernel or a standard tool would send SIGTERM to a random process. What can you tell us about what the program is doing and how it's started? How did you find out about the program's exit status? Can you reproduce the problem? Do you have logs you can check?

[michelemarcon]

It is reading and writing to a serial line, and is responding to UDP and TCP requests. I have wrapped the execution on a bash script and therefore I know the exit code.

[XGouchet]

Posix documentation indicates SIGTERM is strictly a user level event. Is it possible someone else was able to kill your server program?

[michelemarcon]

Very unlikely; however, all I know is the return code (0) which is the same that I get when I terminate the process with ^C (SIGTERM). Maybe I'm overlooking something?

[Gilles 'SO- stop being evil']

You are! The return code 0 means a normal exit. If there was a SIGTERM, $? would be set to 143 (128 + signal number).

[flarn2006]

Also, ^C is SIGINT, not SIGTERM, and that would exit with a code of 130.

[Arsen Karapetjan]

but why does SIGTERM happen? how do I debug why my python script is exiting with that?

[codeforester]

Nice explanation of how exit status is determined upon receiving a signal. However, this answer doesn't address OP's question.

[Gilles 'SO- stop being evil']

@codeforester I answered the question in the body, not the question in the title. Well, one of the questions in the body — given that it was based on a misunderstanding, it is a bit messy. I'll add a few words about the rest.

[Stéphane Chazelas]

Note that using a value above 256 a la ksh is not necessarily better as that prevents it being passed to exit. The yash approach is a good compromise, but see rc for another one. See also Default exit code when process is terminated?

[Arsen Karapetjan]

but why does SIGTERM happen? how do I debug why my python script is exiting with that?

[Arsen Karapetjan]

but why does SIGTERM happen? how do I debug why my python script is exiting with that?

[Stéphane Chazelas]

@CharlieParker, you'd need to determine what process is sending that SIGTERM to the process running the python interpreter that interprets your script. System logs may have some information. If on Linux, you could use auditd/auditctl to log invocations of kill/tkill/tgkill system calls.

[Gilles 'SO- stop being evil']

@CharlieParker SIGTERM happens because some process sends it. It is not sent automatically by the kernel. Something on your system is explicitly calling kill(…, SIGTERM).

[Arsen Karapetjan]

@Gilles'SO-stopbeingevil' could it be that the HPC/cluster is killing my processes because I am taking to much memory? (I am trying to run 112 processes in parallel and do machine learning on it). Thanks in advance for the info. I am for sure (me) not doing any killing, it must be something either in python or the cluster would be my guess.

[Arsen Karapetjan]

could it be that the HPC/cluster is killing my processes because I am taking to much memory? (I am trying to run 112 processes in parallel and do machine learning on it). Thanks in advance for the info. I am for sure (me) not doing any killing, it must be something either in python or the cluster would be my guess.

[Gilles 'SO- stop being evil']

@CharlieParker It's plausible that a HPC cluster would monitor processes and kill those that it considers to have overused resources. You'd have to look at the cluster's documentation or ask the administrators.

[Silicomancer]

What does 'in time' mean, usually? How much time is there between SIGTERM and SIGKILL on shutdown? Milliseconds? Seconds? Minutes?