python multiprocessing.Process.terminate - How to kill child processes

11,033

EDIT: As @svalorzen pointed out in comments this doesn't really work since the reference to self.subprocess is lost.


Finally came to a clean, acceptable solution. Since mp.Process.terminate is a method, we can override it.

class WorkerProcess(mp.Process):
    def run(self):
        # Simulate long running task
        self.subprocess = subprocess.Popen(['python', '-c', 'import time; time.sleep(1000)'])
        self.code = self.subprocess.wait()

    # HERE
    def terminate(self):
      self.subprocess.terminate()
      super(WorkerProcess, self).terminate()
Share:
11,033
redacted
Author by

redacted

Updated on July 28, 2022

Comments

  • redacted
    redacted over 1 year

    This code:

    import multiprocessing as mp
    from threading import Thread
    import subprocess
    import time
    
    class WorkerProcess(mp.Process):
      def run(self):
          # Simulate long running task
          self.subprocess = subprocess.Popen(['python', '-c', 'import time; time.sleep(1000)'])
          self.code = self.subprocess.wait()
    
    
    class ControlThread(Thread):
      def run():
          jobs = []
          for _ in range(2):
              job = WorkerProcess()
              jobs.append(job)
              job.start()
    
          # wait for a while and then kill jobs
          time.sleep(2)
          for job in jobs:
              job.terminate()
    
    if __name__ == "__main__":
        controller = ControlThread()
        controller.start()
    

    When I terminate the spawned WorkerProcess instances. They die just fine, however the subprocesses python -c 'import time; time.sleep(1000) runs until completition. This is well documented in the official docs, but how do I kill the child processes of a killed process?

    A possbile soultion might be:

    • Wrap WorkerProcess.run() method inside try/except block catching SIGTERM, and terminating the subprocess.call call. But I am not sure how to catch the SIGTERM in the WorkerProcess

    • I also tried setting signal.signal(signal.SIGINT, handler) in the WorkerProcess, but I am getting ValuError, because it is allowed to be set only in the main thread.

    What do I do now?