python subclassing multiprocessing.Process
Solution 1
Subclassing multiprocessing.Process
:
However I cannot get back the values, how can I use queues in this way?
Process needs a Queue()
to receive the results... An example of how to subclass multiprocessing.Process
follows...
from multiprocessing import Process, Queue
class Processor(Process):
def __init__(self, queue, idx, **kwargs):
super(Processor, self).__init__()
self.queue = queue
self.idx = idx
self.kwargs = kwargs
def run(self):
"""Build some CPU-intensive tasks to run via multiprocessing here."""
hash(self.kwargs) # Shameless usage of CPU for no gain...
## Return some information back through multiprocessing.Queue
## NOTE: self.name is an attribute of multiprocessing.Process
self.queue.put("Process idx={0} is called '{1}'".format(self.idx, self.name))
if __name__ == "__main__":
NUMBER_OF_PROCESSES = 5
## Create a list to hold running Processor object instances...
processes = list()
q = Queue() # Build a single queue to send to all process objects...
for i in range(0, NUMBER_OF_PROCESSES):
p=Processor(queue=q, idx=i)
p.start()
processes.append(p)
# Incorporating ideas from this answer, below...
# https://stackoverflow.com/a/42137966/667301
[proc.join() for proc in processes]
while not q.empty():
print "RESULT: {0}".format(q.get()) # get results from the queue...
On my machine, this results in...
$ python test.py
RESULT: Process idx=0 is called 'Processor-1'
RESULT: Process idx=4 is called 'Processor-5'
RESULT: Process idx=3 is called 'Processor-4'
RESULT: Process idx=1 is called 'Processor-2'
RESULT: Process idx=2 is called 'Processor-3'
$
Using multiprocessing.Pool
:
FWIW, one disadvantage I've found to subclassing multiprocessing.Process
is that you can't leverage all the built-in goodness of multiprocessing.Pool
; Pool
gives you a very nice API if you don't need your producer and consumer code to talk to each other through a queue.
You can do a lot just with some creative return values... in the following example, I use a dict()
to encapsulate input and output values from pool_job()
...
from multiprocessing import Pool
def pool_job(input_val=0):
# FYI, multiprocessing.Pool can't guarantee that it keeps inputs ordered correctly
# dict format is {input: output}...
return {'pool_job(input_val={0})'.format(input_val): int(input_val)*12}
pool = Pool(5) # Use 5 multiprocessing processes to handle jobs...
results = pool.map(pool_job, xrange(0, 12)) # map xrange(0, 12) into pool_job()
print results
This results in:
[
{'pool_job(input_val=0)': 0},
{'pool_job(input_val=1)': 12},
{'pool_job(input_val=2)': 24},
{'pool_job(input_val=3)': 36},
{'pool_job(input_val=4)': 48},
{'pool_job(input_val=5)': 60},
{'pool_job(input_val=6)': 72},
{'pool_job(input_val=7)': 84},
{'pool_job(input_val=8)': 96},
{'pool_job(input_val=9)': 108},
{'pool_job(input_val=10)': 120},
{'pool_job(input_val=11)': 132}
]
Obviously there are plenty of other improvements to be made in pool_job()
, such as error handling, but this illustrates the essentials. FYI, this answer provides another example of how to use multiprocessing.Pool
.
Solution 2
The return value of Process.run
doesn't go anywhere. You need to send them back to the parent process, e.g. using a multiprocessing.Queue
(docs here).
Solution 3
Mike's answer is the best, but just for completeness I want to mention that I prefer harvesting the queue out of join
contexts so the last bit would look like this:
[proc.join() for proc in processes] # 1. join
while not q.empty(): # 2. get the results
print "RESULT: %s" % q.get()
Solution 4
Thanks a lot everyone.
Now heres how i got it done :)
In this example i use multiple queus as i do not want to communicate between each ohter but only with parent process.
from multiprocessing import Process,Queue
class Processor(Process):
def __init__(self,queue):
Process.__init__(self)
self.que=queue
def get_name(self):
return "Process %s" % self.name
def run(self):
self.que.put(self.get_name())
if __name__ == "__main__":
processes = []
for i in range(0,5):
p=Processor(Queue())
processes.append(p)
p.start()
for p in processes:
p.join()
print p.que.get()
Comments
-
Phyo Arkar Lwin almost 4 years
I am new to python object oriented and I am rewriting my existing application as an object oriented version, because now developers are increasing and my code is becoming un-maintainable.
Normally I use multiprocessing queues but I found from this example http://www.doughellmann.com/PyMOTW/multiprocessing/basics.html that I can subclass
multiprocessing.Process
so I think it's a good idea and I wrote a class to test like this:code:
from multiprocessing import Process class Processor(Process): def return_name(self): return "Process %s" % self.name def run(self): return self.return_name() processes = [] if __name__ == "__main__": for i in range(0,5): p=Processor() processes.append(p) p.start() for p in processes: p.join()
However I cannot get back the values, how can I use queues in this way?
EDIT: I want to get the return value and thinking where to put
Queues()
. -
Phyo Arkar Lwin over 12 yearsSo , in one of the method have to accept Queue object as parameter right?
-
Phyo Arkar Lwin over 12 yearsPlease review my code and let me know what i can improve to be more pythonic / and better practice.
-
Phyo Arkar Lwin over 12 yearsDone! i created an init method accepting queues. this in-turn extends multiprocessing.Process to accept Queues directly :)
-
Phyo Arkar Lwin over 12 yearsTHanks i will check. But i've read supers are dangerous especially multiple inheritance? is that true?
-
Phyo Arkar Lwin over 12 yearsThanks for the correction. This code
return self.queue.put(self.return_name())
returns a queue? -
Mike Pennington over 12 yearsthe return value from the
Processor()
call itself is irrelevant; you are passing the return value through theQueue()
instance that you started theProcessor()
with -
Mike Pennington over 12 yearsyou need to be very careful when using
super()
with multiple inheritance. That said, I don't see multiple inheritance in the question you asked -
Thomas K over 12 yearsYou don't actually need a queue for each process. Multiple processes can put things into one queue without a problem. This is common when you have worker processes doing some computation, and putting output in a common result queue.
-
Gerald almost 3 yearswith the super().__init__() method, do you pass in the target/arguments here, or in the self.run method?
-
Verthais almost 3 yearsDoes it really runs in parallel or does it get stuck on GIL while trying to access shared resource ( which in this case is our
Queue
) ?