How do you share data between a parent and forked child process in Python?

Question

I'm pretty sure one would do this using the os.plock(op) function, but I have no idea how. Also, if there's a better way, I'd be grateful to find out. Code snippets are very welcome.

Solution 1

Subprocess replaces os.popen, os.system, os.spawn, popen2 and commands. A simple example for piping would be:

p1 = Popen(["dmesg"], stdout=PIPE)
p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE)
output = p2.communicate()[0]

You could also use a memory mapped file with the flag=MAP_SHARED for shared memory between processes.

multiprocessing abstracts both pipes and shared memory and provides a higher level interface. Taken from the Processing documentation:

from multiprocessing import Process, Pipe

def f(conn):
    conn.send([42, None, 'hello'])
    conn.close()

if __name__ == '__main__':
    parent_conn, child_conn = Pipe()
    p = Process(target=f, args=(child_conn,))
    p.start()
    print parent_conn.recv()   # prints "[42, None, 'hello']"
    p.join()

Solution 2

Take a look at the multiprocessing module new in python 2.6 (also available for earlier versions a pyprocessing

Here's an example from the docs illustrating passing information using a pipe for instance:

from multiprocessing import Process, Pipe

def f(conn):
    conn.send([42, None, 'hello'])
    conn.close()

if __name__ == '__main__':
    parent_conn, child_conn = Pipe()
    p = Process(target=f, args=(child_conn,))
    p.start()
    print parent_conn.recv()   # prints "[42, None, 'hello']"
    p.join()

Solution 3

This is pretty much Python-independent! It's a classic example of Unix interprocess communication. One good option is to use popen() to open a pipe between the parent and child processes, and pass data/messages back and forth along the pipe.

Take a look at the subprocess module, which can set up the necessary pipes automatically while spawning child processes.

Solution 4

You have two options: os.popen* in the os module, or you can use the subprocess module to the same effect. The Python manual has pretty documentation and examples for popen and subprocess.

Solution 5

If you are doing low-level operating system forking and really want to avoid using pipes, it is possible to use shared memory-mapped files as well. This is not nearly as nice as using subprocess or popen pipes, but including the answer for completeness...

There's a full example here, but basically you can combine the os file handling and mmap modules:

import mmap, os, tempfile
fd, tmpfile = tempfile.mkstemp()
os.write(fd, '\x00' * mmap.PAGESIZE)
os.lseek(fd, 0, os.SEEK_SET)
child_pid = os.fork()
if child_pid:
    buf = mmap.mmap(fd, mmap.PAGESIZE, mmap.MAP_SHARED, mmap.PROT_READ)
    os.waitpid(child_pid, 0)
    child_message = buf.readline()
    print(child_message)
    os.close(fd)
else:
    buf = mmap.mmap(fd, mmap.PAGESIZE, mmap.MAP_SHARED, mmap.PROT_WRITE)
    buf.write('testing\n')
    os.close(fd)

View more solutions