Multiprocessing - Shared Array
Since you're only returning state from the child process to the parent process, then using a shared array and explicity locks is overkill. You can use Pool.map
or Pool.starmap
to accomplish exactly what you need. For example:
from multiprocessing import Pool
class Adder:
"""I'm using this class in place of a monte carlo simulator"""
def add(self, a, b):
return a + b
def setup(x, y, z):
"""Sets up the worker processes of the pool.
Here, x, y, and z would be your global settings. They are only included
as an example of how to pass args to setup. In this program they would
be "some arg", "another" and 2
"""
global adder
adder = Adder()
def job(a, b):
"""wrapper function to start the job in the child process"""
return adder.add(a, b)
if __name__ == "__main__":
args = list(zip(range(10), range(10, 20)))
# args == [(0, 10), (1, 11), ..., (8, 18), (9, 19)]
with Pool(initializer=setup, initargs=["some arg", "another", 2]) as pool:
# runs jobs in parallel and returns when all are complete
results = pool.starmap(job, args)
print(results) # prints [10, 12, ..., 26, 28]
Vedant7
Updated on June 15, 2022Comments
-
Vedant7 about 2 years
So I'm trying to implement multiprocessing in python where I wish to have a Pool of 4-5 processes running a method in parallel. The purpose of this is to run a total of thousand Monte simulations (250-200 simulations per process) instead of running 1000. I want each process to write to a common shared array by acquiring a lock on it as soon as its done processing the result for one simulation, writing the result and releasing the lock. So it should be a three step process :
- Acquire lock
- Write result
- Release lock for other processes waiting to write to array.
Everytime I pass the array to the processes each process creates a copy of that array which I donot want as I want a common array. Can anyone help me with this by providing sample code?
-
mata almost 8 yearsThat is not a shared array. Changing it in a subprocess won't have any effect in the parent.
-
mata almost 8 yearsAlso, an Array is created by default with a lock attached which you can get using its
get_lock()
method, no need to allocate it explicitly unless you want to use a different lock type. The body could then become justwith array.get_lock(): ...
-
RodrikTheReader over 6 years@mata I need some clarification. Do you mean that I can change the function f to just "def f(array): with array.get_lock(): #modify array here# " and there is no need to create a lock object and explicitly pass it?
-
Julien over 6 years@RodrikTheReader yes this is what mata explains: docs.python.org/2/library/…
-
RodrikTheReader over 6 years@Julien Thanks. I am actually struggling with a similar shared memory problem as posted in the question by OP. I need 4 separate arrays that can be shared by multiple processes. I've defined the shared array in a class and I'm passing the objects of that class to a function which writes on the shared arrays. I'm spawning multiple processes and the target is this function. Anyway, the program is not working as I expect. Do you think instead of creating objects, I should just create the shared arrays in the main program and pass them to the function?
-
Julien over 6 years@RodrikTheReader sorry but I do not have enough information. You may continue to search a bit by yourself and then ask a new question.
-
RodrikTheReader over 6 years@Julien No problem