How do I run another script in Python without waiting for it to finish?

Question

I am creating a little dashboard for a user that will allow him to run specific jobs. I am using Django so I want him to be able to click a link to start the job and then return the page back to him with a message that the job is running. The results of the job will be emailed to him later.

I believe I am supposed to use subprocess.Popen but I'm not sure of that. So in pseudocode, here is what I want to do:

if job == 1:
    run script in background: /path/to/script.py
    return 'Job is running'

Solution 1

p = subprocess.Popen([sys.executable, '/path/to/script.py'], 
                                    stdout=subprocess.PIPE, 
                                    stderr=subprocess.STDOUT)

That will start the subprocess in background. Your script will keep running normally.

Read the documentation here.

Solution 2

subprocess.Popen is indeed what you are looking for.

Solution 3

Although if you find that you want to start communicating a bunch of information between the subprocess and the parent, you may want to consider a thread, or RPC framework like Twisted.

But most likely those are too heavy for your application.

Comments

[sheats]

nosklo: Thanks. How would I pass in arguments to the script?

[Devin Jeanpierre]

Additional elements in the list passed as the first argument. The linked documentation is useful, and linked to for a reason.

[Bodo Thiesen]

In a non-windows environment, you could check out os.fork as well

[sheats]

Devin: Thanks. I ask on here because it was hard for me to figure that out from the docs. I don't see sys.executable anywhere in the docs. If all we ever needed was docs, we wouldn't need SO =)

[nosklo]

@sheats: sys.executable is documented here docs.python.org/library/sys.html#sys.executable

[akaihola]

@nosklo Is this a good thing to do in the context of a web server? I've seen advice to run a separate service for long-running jobs and communicate with a queue mechanism.

[nosklo]

@akaihola, it seems to be the original question. sheats is asking for a way to run a separate process (outside the webserver process). A communication mechanism was not part of the question.

[jfs]

stdout=PIPE might hang the subprocess, redirect to DEVNULL to discard the output.

[user5319825]

@nosklo can we pass a new parameter (just like we pass to a function) from previous process to the new sub-process?