changing global variable when multiprocessing in python
14,967
The worker processes spawned by the pool get their own copy of the global variable and update that. They don't share memory unless you set that up explicitly. The easiest solution is to communicate the final value of test
back to the main process, e.g. via the return value. Something like (untested):
def pro(temp_line):
test = 0
temp_line = temp_line.strip().split()
test = test + 1
return test, len(temp_line)
if __name__ == "__main__":
with open("somefile.txt") as lines:
pool = mp.Pool(processes = 10)
tests_and_t = pool.map(pro,lines.readlines())
tests, t = zip(*test_and_t)
test = sum(tests)
Author by
user1423020
Updated on June 04, 2022Comments
-
user1423020 almost 2 years
So what I am trying to do ultimately is read a line, do some calculations with the info in that line, then add the result to some global object, but I can never seem to get it to work. For instance, test is always 0 in the code below. I know this is wrong, and I have tried doing it other ways, but it still isn't working.
import multiprocessing as mp File = 'HGDP_FinalReport_Forward.txt' #short_file = open(File) test = 0 def pro(temp_line): global test temp_line = temp_line.strip().split() test = test + 1 return len(temp_line) if __name__ == "__main__": with open("HGDP_FinalReport_Forward.txt") as lines: pool = mp.Pool(processes = 10) t = pool.map(pro,lines.readlines())