run multiple instances of python script simultaneously

Question

I am trying to create 86 instances of task.py to run simultaneously.

import sys
import subprocess

for file in range(86):
    subprocess.call([sys.executable,'task.py',str(file)+'in.csv',str(filen)+'out.csv'])

Solution

subprocess.call waits for command to complete. Use subprocess.Popen instead:

import sys
import subprocess

procs = []
for i in range(86):
    proc = subprocess.Popen([sys.executable, 'task.py', '{}in.csv'.format(i), '{}out.csv'.format(i)])
    procs.append(proc)

for proc in procs:
    proc.wait()

Comments

[Alex]

What is the problem?

[user14372]

@Alex the problem was "subprocess.call waits for command to complete. Use subprocess.Popen instead:"

[Lisa]

Is this solution a "multithread" solution? Or actually it launches multiple instance of python?

[falsetru]

@Lisa, As the name of the module suggest, it launches multiple instances (sub"process"). (OP's original code also launches multiple instances)

[Pawankumar Dubey]

I was just searching for something like this for so much time. Thank you

[MasayoMusic]

I can't get it to work when looping and appending: out =subprocess.Popen("hugo server -D", stdout = subprocess.PIPE) outputs.append(out) And then iterating through outputs and calling output.wait(). It opens up the first program, but then doesn't open up the next one unless I close the first program.

[falsetru]

@MasayoMusic, Replace "hugo server -D" -> ['hugo', 'server', '-D']. If that does not work, please post a separate question so that others can answer you.

[MasayoMusic]

@falsetru I did create one yesterday (just updated), but I couldn't get an answer so I have been hunting around: https://stackoverflow.com/questions/57466534/cant-launch-mul‌​tiple-local-servers-‌​using-subprocess-in-‌​python

[Pe Dro]

@falsetru What is the purpose of the sys.executable in the Popen command...I looked into the official docs but could not find them using it.

[falsetru]

@PeDro, sys.executable is an absolute path to the executable of current python process. It was coming from OP's code. My assumption is that OP wanted to use the same python executable for the sub-process.