Python Multiprocessing Kill Processes

Question

I am learning how to use the Python multiprocessing library. However, while I am going through some of the examples, I ended up with many python processes running in my background.

One of the example looks like below:

from multiprocessing import Process, Lock

def f(l, i):
    l.acquire()
    print 'hello world', i
    l.release()

if __name__ == '__main__':
    lock = Lock()

    for num in range(10):   # I changed the number of iterations from 10 to 1000...
        Process(target=f, args=(lock, num)).start()

Now here is a screen shot of my 'TOP' command:

88950  Python       0.0  00:00.00 1    0    9     91    1584K  5856K  2320K  1720K  2383M  82441 1     sleeping 1755113321 799
88949  Python       0.0  00:00.00 1    0    9     91    1584K  5856K  2320K  1720K  2383M  82441 1     sleeping 1755113321 798
88948  Python       0.0  00:00.00 1    0    9     91    1580K  5856K  2316K  1716K  2383M  82441 1     sleeping 1755113321 797
88947  Python       0.0  00:00.00 1    0    9     91    1580K  5856K  2316K  1716K  2383M  82441 1     sleeping 1755113321 796
88946  Python       0.0  00:00.00 1    0    9     91    1576K  5856K  2312K  1712K  2383M  82441 1     sleeping 1755113321 795
88945  Python       0.0  00:00.00 1    0    9     91    1576K  5856K  2312K  1712K  2383M  82441 1     sleeping 1755113321 794
88944  Python       0.0  00:00.00 1    0    9     91    1576K  5856K  2312K  1712K  2383M  82441 1     sleeping 1755113321 794
88943  Python       0.0  00:00.00 1    0    9     91    1572K  5856K  2308K  1708K  2383M  82441 1     sleeping 1755113321 792
88942  Python       0.0  00:00.00 1    0    9     91    1568K  5856K  2304K  1708K  2383M  82441 1     sleeping 1755113321 790
88941  Python       0.0  00:00.00 1    0    9     91    1564K  5856K  2300K  1704K  2383M  82441 1     sleeping 1755113321 789
88938  Python       0.0  00:00.00 1    0    9     91    1564K  5856K  2300K  1704K  2383M  82441 1     sleeping 1755113321 788
88936  Python       0.0  00:00.00 1    0    9     91    1576K  5856K  2296K  1716K  2383M  82441 1     sleeping 1755113321 787
88935  Python       0.0  00:00.00 1    0    9     91    1560K  5856K  2296K  1700K  2383M  82441 1     sleeping 1755113321 787
88934  Python       0.0  00:00.00 1    0    9     91    1560K  5856K  2296K  1700K  2383M  82441 1     sleeping 1755113321 786
88933  Python       0.0  00:00.00 1    0    9     91    1556K  5856K  2292K  1696K  2383M  82441 1     sleeping 1755113321 785
88932  Python       0.0  00:00.00 1    0    9     91    1556K  5856K  2292K  1696K  2383M  82441 1     sleeping 1755113321 784
88931  Python       0.0  00:00.00 1    0    9     91    1552K  5856K  2288K  1692K  2383M  82441 1     sleeping 1755113321 783
88930  Python       0.0  00:00.00 1    0    9     91    1612K  5856K  2288K  1752K  2383M  82441 1     sleeping 1755113321 783
88929  Python       0.0  00:00.00 1    0    9     91    1588K  5856K  2288K  1728K  2383M  82441 1     sleeping 1755113321 782
88927  Python       0.0  00:00.00 1    0    9     91    1608K  5856K  2284K  1748K  2383M  82441 1     sleeping 1755113321 781
88926  Python       0.0  00:00.00 1    0    9     91    1548K  5856K  2284K  1688K  2383M  82441 1     sleeping 1755113321 780
88924  Python       0.0  00:00.00 1    0    9     91    1556K  5856K  2276K  1700K  2383M  82441 1     sleeping 1755113321 778
88923  Python       0.0  00:00.00 1    0    9     91    1540K  5856K  2276K  1684K  2383M  82441 1     sleeping 1755113321 777
88922  Python       0.0  00:00.00 1    0    9     91    1540K  5856K  2276K  1684K  2383M  82441 1     sleeping 1755113321 776
88921  Python       0.0  00:00.00 1    0    9     91    1536K  5856K  2272K  1680K  2383M  82441 1     sleeping 1755113321 774
88920  Python       0.0  00:00.00 1    0    9     91    1528K  5856K  2264K  1672K  2383M  82441 1     sleeping 1755113321 771
88919  Python       0.0  00:00.00 1    0    9     91    1528K  5856K  2264K  1672K  2383M  82441 1     sleeping 1755113321 771
88918  Python       0.0  00:00.00 1    0    9     91    1528K  5856K  2264K  1672K  2383M  82441 1     sleeping 1755113321 770
....

1. I don't know how to kill them in one go.

   ps ... | grep python .... kill?

2. what kind of python code do I need to add to avoid this miserable situation again. Thanks!

Solution 1

You need to .join() on your processes in a worker Queue, which will lock them to the calling application until all of them succeed or kill when the parent is killed, and run them in daemon mode.

http://forums.xkcd.com/viewtopic.php?f=11&t=94726

end daemon processes with multiprocessing module

http://docs.python.org/2/library/multiprocessing.html#the-process-class

http://www.python.org/dev/peps/pep-3143/#correct-daemon-behaviour

Solution 2

The answer pointed by Blake VandeMerwe is listed and explained below hope could be helpful for other users:

Original Author:

kill -9 `ps -ef | grep test.py | grep -v grep | awk '{print $2}'`

Explaination:

1. "ps -ef": show all the processes including those without controlling terminals, which are exactly the countless processes generated by MULTIPROCESSING library.

2. "grep test.py": find all the processes which are generated by this script, which is the name of my python script.

3. "grep -v grep": excluded the grep operation itself from the 'killing list'

4. "awk '{print $2}'": using AWK to separate every single records into row and print out the second row which in this case, are the process id colum.

5. "kill -9" is force kill process, arguments should be UID. The complete output of previous steps are put together by "`", which is the character on the left of number 1 on regular keyboard. which treat them as a variable and pass the value to kill.

Comments

[B.Mr.W.]

The is 'you should have...' code to avoid this happen again, right? do you know how to kill all the processes generated by multiprocessing?

[Blacklight Shining]

-1. No! Don't just SIGKILL first!