How to process SIGTERM signal gracefully?
Solution 1
A class based clean to use solution:
import signal
import time
class GracefulKiller:
kill_now = False
def __init__(self):
signal.signal(signal.SIGINT, self.exit_gracefully)
signal.signal(signal.SIGTERM, self.exit_gracefully)
def exit_gracefully(self, *args):
self.kill_now = True
if __name__ == '__main__':
killer = GracefulKiller()
while not killer.kill_now:
time.sleep(1)
print("doing something in a loop ...")
print("End of the program. I was killed gracefully :)")
Solution 2
First, I'm not certain that you need a second thread to set the shutdown_flag
.
Why not set it directly in the SIGTERM handler?
An alternative is to raise an exception from the SIGTERM
handler, which will be propagated up the stack. Assuming you've got proper exception handling (e.g. with with
/contextmanager
and try: ... finally:
blocks) this should be a fairly graceful shutdown, similar to if you were to Ctrl+C your program.
Example program signals-test.py
:
#!/usr/bin/python
from time import sleep
import signal
import sys
def sigterm_handler(_signo, _stack_frame):
# Raises SystemExit(0):
sys.exit(0)
if sys.argv[1] == "handle_signal":
signal.signal(signal.SIGTERM, sigterm_handler)
try:
print "Hello"
i = 0
while True:
i += 1
print "Iteration #%i" % i
sleep(1)
finally:
print "Goodbye"
Now see the Ctrl+C behaviour:
$ ./signals-test.py default
Hello
Iteration #1
Iteration #2
Iteration #3
Iteration #4
^CGoodbye
Traceback (most recent call last):
File "./signals-test.py", line 21, in <module>
sleep(1)
KeyboardInterrupt
$ echo $?
1
This time I send it SIGTERM
after 4 iterations with kill $(ps aux | grep signals-test | awk '/python/ {print $2}')
:
$ ./signals-test.py default
Hello
Iteration #1
Iteration #2
Iteration #3
Iteration #4
Terminated
$ echo $?
143
This time I enable my custom SIGTERM
handler and send it SIGTERM
:
$ ./signals-test.py handle_signal
Hello
Iteration #1
Iteration #2
Iteration #3
Iteration #4
Goodbye
$ echo $?
0
Solution 3
Here is a simple example without threads or classes.
import signal
run = True
def handler_stop_signals(signum, frame):
global run
run = False
signal.signal(signal.SIGINT, handler_stop_signals)
signal.signal(signal.SIGTERM, handler_stop_signals)
while run:
pass # do stuff including other IO stuff
Solution 4
I think you are near to a possible solution.
Execute mainloop
in a separate thread and extend it with the property shutdown_flag
. The signal can be caught with signal.signal(signal.SIGTERM, handler)
in the main thread (not in a separate thread). The signal handler should set shutdown_flag
to True and wait for the thread to end with thread.join()
Solution 5
Based on the previous answers, I have created a context manager which protects from sigint and sigterm.
import logging
import signal
import sys
class TerminateProtected:
""" Protect a piece of code from being killed by SIGINT or SIGTERM.
It can still be killed by a force kill.
Example:
with TerminateProtected():
run_func_1()
run_func_2()
Both functions will be executed even if a sigterm or sigkill has been received.
"""
killed = False
def _handler(self, signum, frame):
logging.error("Received SIGINT or SIGTERM! Finishing this block, then exiting.")
self.killed = True
def __enter__(self):
self.old_sigint = signal.signal(signal.SIGINT, self._handler)
self.old_sigterm = signal.signal(signal.SIGTERM, self._handler)
def __exit__(self, type, value, traceback):
if self.killed:
sys.exit(0)
signal.signal(signal.SIGINT, self.old_sigint)
signal.signal(signal.SIGTERM, self.old_sigterm)
if __name__ == '__main__':
print("Try pressing ctrl+c while the sleep is running!")
from time import sleep
with TerminateProtected():
sleep(10)
print("Finished anyway!")
print("This only prints if there was no sigint or sigterm")
Related videos on Youtube
zerkms
Yet another software developer https://cv.zerkms.com/ #SOreadytohelp
Updated on September 02, 2021Comments
-
zerkms almost 3 years
Let's assume we have such a trivial daemon written in python:
def mainloop(): while True: # 1. do # 2. some # 3. important # 4. job # 5. sleep mainloop()
and we daemonize it using
start-stop-daemon
which by default sendsSIGTERM
(TERM
) signal on--stop
.Let's suppose the current step performed is
#2
. And at this very moment we're sendingTERM
signal.What happens is that the execution terminates immediately.
I've found that I can handle the signal event using
signal.signal(signal.SIGTERM, handler)
but the thing is that it still interrupts the current execution and passes the control tohandler
.So, my question is - is it possible to not interrupt the current execution but handle the
TERM
signal in a separated thread (?) so that I was able to setshutdown_flag = True
so thatmainloop()
had a chance to stop gracefully?-
Eric Urban almost 11 yearsI did what you are asking for before by using
signalfd
and masking out the delivery of theSIGTERM
to the process.
-
-
zerkms almost 11 yearsYep, a separated thread is how I've finally solved it, thanks
-
zerkms about 10 years"Why not set it directly in the SIGTERM handler" --- because the worker thread would interrupt on a random place. If you put multiple statements into your worker loop you will see that your solution terminates a worker on a random position, which leaves the job in an unknown state.
-
Marian about 9 yearsWorks well for me, also in a Docker context. Thanks!
-
Suor about 9 yearsIf you just set a flag and not raise exception then it will be the same as with thread. So using thread is superfluous here.
-
Mayank Jaiswal almost 9 yearsThreads are not required here. In a single threaded program itself, you can first register a signal handler (registering a signal handler is non blocking) and then write mainloop. Signal handler function should set a flag when and loop should check for this flag. I have pasted a class based solution for the same here.
-
rsaw almost 9 yearsThanks for the idea! I used a modified approach in reboot-guard. github.com/ryran/reboot-guard/blob/master/rguard#L284:L304
-
jose.angel.jimenez over 8 yearsThis is the best answer (no threads required), and should be the preferred first-try approach.
-
Mausy5043 over 8 years@mayank-jaiswal : This is brilliant! Thanks! Just to be sure: shouldn't it be
class GracefulKiller()
? -
Mayank Jaiswal over 8 years@Mausy5043 Python allows you to not have parenthesis for defining classes. Although it's perfectly fine for python 3.x, but for python 2.x, best practice is to use "class XYZ(object):". Reason being: docs.python.org/2/reference/datamodel.html#newstyle
-
Admin almost 8 yearsNo way that having a second thread is necessary. Register signal handler.
-
Admin almost 8 years@MayankJaiswal good answer. If I want to use this class to catch USR1 signal for log rotation, how can I return the file object to the outter scope?
-
chrisfauerbach over 7 yearsFollow up, to keep you motivated, thank you. I use this all the time.
-
tripleee over 7 yearsThe
import sys
is harmless, but doesn't do anything useful here. (Chances are other parts of your code will usesys
anyway, but not this code.) -
Mayank Jaiswal over 7 yearsdeleted that
import sys
-
Kamil Sindi about 7 yearshelpful page: g-loaded.eu/2016/11/24/…
-
igonejack over 6 yearsWould it be atomic, I mean it looks like multi threads in risk of reading and writing the same memory in same time.
-
Alceste_ almost 6 yearsIn worse case, that would simply mean doing another iteration before shuting down gracefully. The
False
value is set only once, and then it can only go from False to True so multiple access is not an issue. -
Steve almost 5 yearsIf the sleep() is long it will be a long wait before it breaks out of the loop. Check out stackoverflow.com/a/46346184/418819 for a solution that will exit immediately.
-
M Y over 3 yearsI think it needs to be
GracefulKiller.kill_now = True
and notself.kill_now = True
-
Vyacheslav Napadovsky over 3 yearsIt seems it won't support nested
with TerminateProtected():
statements. -
Alex Peters about 3 years@VyacheslavNapadovsky, it retains the previous handlers on entry and restores them on (context) exit, so it seems okay to me?
-
Vyacheslav Napadovsky about 3 years@AlexPeters, yes, but you don't call the original handler when event occurs, thus any handlers will get ignored. And with addition to that that we have exception handling mechanism that may catch and ignore exception thrown by sys.exit(0), nested
with
statements with exception handlers in-between will lead to ignoring the termination request. (And make you hit ctrl+C) several times at least. -
user5305519 about 3 yearsNot too sure if I am doing it right but following this code and pressing
CTRL
+C
givesTypeError: exit_gracefully() takes 1 positional argument but 3 were given
. I solved it by changingdef exit_gracefully(self)
todef exit_gracefully(self, sig, frame)
per stackoverflow.com/a/1112350/5305519 -
Mattias Wallin almost 3 yearsThe question was how to gracefully quit on SIGTERM. Your example shows how to ignore SIGINT. You can change to ignore SIGTERM, but it is a bad idea for a daemon to ignore it, because it will then instead be SIGKILL:ed after a timeout, and that can't be ignored or gracefully handled at all.
-
Mattias Wallin almost 3 yearsYou generally can not do resource cleanup in a signal handler, since it can't know what the program was doing when receiving the signal. Sure it have the stacktrace, but that is almost never enough to do something useful.
-
loretoparisi almost 3 yearsIn my case the cleanup is related to running program code, so I exactly know which resources to safely close (like db connections, pending sockets and IO, etc.) while regarding external processes, just in my case, I have piped programs (opened to input on stdin) that I can even close at signal handler because I have a reference to each of them. But of course this will only work with this approach or similar.
-
Mattias Wallin almost 3 yearsHow do
exit_gracefully()
know which DB connections are safe to close? How do it wait for it to be safe? -
loretoparisi almost 3 yearsyes this is application logic specific, if you have good object wrappers, you can do it, but of course is application specific, and - my two cents - there is no general solution! thanks for your comments.
-
Zahra Ashouri over 2 yearsI think there is a risk of changing signal handlers from default to another handler. isn't it necessary to set back the previous default handlers for those signals?