How to terminate multiprocessing Pool processes?

Question

I'm working on a renderfarm, and I need my clients to be able to launch multiple instances of a renderer, without blocking so the client can receive new commands. I've got that working correctly, however I'm having trouble terminating the created processes.

At the global level, I define my pool (so that I can access it from any function):

p = Pool(2)

I then call my renderer with apply_async:

for i in range(totalInstances):
    p.apply_async(render, (allRenderArgs[i],args[2]), callback=renderFinished)
p.close()

That function finishes, launches the processes in the background, and waits for new commands. I've made a simple command that will kill the client and stop the renders:

def close():
    '''
        close this client instance
    '''
    tn.write ("say "+USER+" is leaving the farm\r\n")
    try:
        p.terminate()
    except Exception,e:
        print str(e)
        sys.exit()

It doesn't seem to give an error (it would print the error), the python terminates but the background processes are still running. Can anyone recommend a better way of controlling these launched programs?

Solution 1

Found the answer to my own question. The primary problem was that I was calling a third-party application rather than a function. When I call the subprocess [either using call() or Popen()] it creates a new instance of python whose only purpose is to call the new application. However when python exits, it will kill this new instance of python and leave the application running.

The solution is to do it the hard way, by finding the pid of the python process that is created, getting the children of that pid, and killing them. This code is specific for osx; there is simpler code (that doesn't rely on grep) available for linux.

for process in pool:
    processId = process.pid
    print "attempting to terminate "+str(processId)
    command = " ps -o pid,ppid -ax | grep "+str(processId)+" | cut -f 1 -d \" \" | tail -1"
    ps_command = Popen(command, shell=True, stdout=PIPE)
    ps_output = ps_command.stdout.read()
    retcode = ps_command.wait()
    assert retcode == 0, "ps command returned %d" % retcode
    print "child process pid: "+ str(ps_output)
    os.kill(int(ps_output), signal.SIGTERM)
    os.kill(int(processId), signal.SIGTERM)

Solution 2

I found solution: stop pool in separate thread, like this:

def close_pool():
    global pool
    pool.close()
    pool.terminate()
    pool.join()

def term(*args,**kwargs):
    sys.stderr.write('\nStopping...')
    # httpd.shutdown()
    stophttp = threading.Thread(target=httpd.shutdown)
    stophttp.start()
    stoppool=threading.Thread(target=close_pool)
    stoppool.daemon=True
    stoppool.start()


signal.signal(signal.SIGTERM, term)
signal.signal(signal.SIGINT, term)
signal.signal(signal.SIGQUIT, term)

Works fine and always i tested.

signal.SIGINT

Interrupt from keyboard (CTRL + C). Default action is to raise KeyboardInterrupt.

signal.SIGKILL

Kill signal. It cannot be caught, blocked, or ignored.

signal.SIGTERM

Termination signal.

signal.SIGQUIT

Quit with core dump.

Solution 3

If you're still experiencing this issue, you could try simulating a Pool with daemonic processes (assuming you are starting the pool/processes from a non-daemonic process). I doubt this is the best solution since it seems like your Pool processes should be exiting, but this is all I could come up with. I don't know what your callback does so I'm not sure where to put it in my example below.

I also suggest trying to create your Pool in __main__ due to my experience (and the docs) with weirdness occurring when processes are spawned globally. This is especially true if you're on Windows: http://docs.python.org/2/library/multiprocessing.html#windows

from multiprocessing import Process, JoinableQueue

# the function for each process in our pool
def pool_func(q):
    while True:
        allRenderArg, otherArg = q.get() # blocks until the queue has an item
        try:
            render(allRenderArg, otherArg)
        finally: q.task_done()

# best practice to go through main for multiprocessing
if __name__=='__main__':
    # create the pool
    pool_size = 2
    pool = []
    q = JoinableQueue()
    for x in range(pool_size):
        pool.append(Process(target=pool_func, args=(q,)))

    # start the pool, making it "daemonic" (the pool should exit when this proc exits)
    for p in pool:
        p.daemon = True
        p.start()

    # submit jobs to the queue
    for i in range(totalInstances):
        q.put((allRenderArgs[i], args[2]))

    # wait for all tasks to complete, then exit
    q.join()

Comments

[schlamar]

Try to enable debug logging with from multiprocessing import util; util.get_logger().setLevel(util.DEBUG) and paste the output.

[mdscruggs]

I've seen behavior like this before but can't reproduce it now...I wonder if calling p.join() would help after calling p.terminate()? I also wonder if you even need to call terminate and if just doing sys.exit() will properly garbage collect the Pool and all of its processes.

[tk421storm]

when I try to enable logging I'm getting this in the console:" No handlers could be found for logger "multiprocessing". Unfortunately, p.join() after p.terminate() doesn't make a difference, and sys.exit() closes the python but leaves the processes running in the background.

[jfs]

try multiprocessing.log_to_stderr().setLevel(logging.DEBUG). Does render() start additional processes e.g., using subprocess module?

[tk421storm]

interesting! nice tip about defining in main instead of globally. I rebuilt this way and it didn't solve my problem (see below) but I like the construction better. Thanks!

[Sabito 錆兎 stands with Ukraine]

What are SIGTERM,SIGINT and SIGQUIT?

[eri]

Its for interupting with ctrl+c, task kill, sys.exit(), etc..