PyInstaller-built Windows EXE fails with multiprocessing

Question

In my project I'm using Python's multiprocessing library to create multiple processes in __main__. The project is being packaged into a single Windows EXE using PyInstaller 2.1.1.

I create new processes like so:

from multiprocessing import Process
from Queue import Empty

def _start():
    while True:
        try:
            command = queue.get_nowait()
        # ... and some more code to actually interpret commands
        except Empty:
            time.sleep(0.015)

def start():
    process = Process(target=_start, args=args)
    process.start()
    return process

And in __main__:

if __name__ == '__main__':
    freeze_support()

    start()

Unfortunately, when packaging the application into an EXE and launching it, I get WindowsError 5 or 6 (seems random) at this line:

command = queue.get_nowait()

A recipe at PyInstaller's homepage claims that I have to modify my code to enable multiprocessing in Windows when packaging the application as a single file.

I'm reproducing the code here:

import multiprocessing.forking
import os
import sys


class _Popen(multiprocessing.forking.Popen):
    def __init__(self, *args, **kw):
        if hasattr(sys, 'frozen'):
            # We have to set original _MEIPASS2 value from sys._MEIPASS
            # to get --onefile mode working.
            # Last character is stripped in C-loader. We have to add
            # '/' or '\\' at the end.
            os.putenv('_MEIPASS2', sys._MEIPASS + os.sep)
        try:
            super(_Popen, self).__init__(*args, **kw)
        finally:
            if hasattr(sys, 'frozen'):
                # On some platforms (e.g. AIX) 'os.unsetenv()' is not
                # available. In those cases we cannot delete the variable
                # but only set it to the empty string. The bootloader
                # can handle this case.
                if hasattr(os, 'unsetenv'):
                    os.unsetenv('_MEIPASS2')
                else:
                    os.putenv('_MEIPASS2', '')


class Process(multiprocessing.Process):
    _Popen = _Popen


class SendeventProcess(Process):
    def __init__(self, resultQueue):
        self.resultQueue = resultQueue

        multiprocessing.Process.__init__(self)
        self.start()

    def run(self):
        print 'SendeventProcess'
        self.resultQueue.put((1, 2))
        print 'SendeventProcess'


if __name__ == '__main__':
    # On Windows calling this function is necessary.
    if sys.platform.startswith('win'):
        multiprocessing.freeze_support()
    print 'main'
    resultQueue = multiprocessing.Queue()
    SendeventProcess(resultQueue)
    print 'main'

My frustration with this "solution" is that, one, it's absolutely unclear what exactly it is patching, and, two, that it's written in such a convoluted way that it becomes impossible to infer which parts are the solution, and which are just an illustration.

Can anyone share some light on this issue, and provide insight what exactly needs to be changed in a project that enables multiprocessing in PyInstaller-built single-file Windows executables?

Solution 1

Answering my own questions after finding this PyInstaller ticket:

Apparently all we have to do is provide a Process (and _Popen) class as shown below, and use it instead of multiprocessing.Process. I've corrected and simplified the class to work on Windows only, *ix systems might need different code.

For the sake of completeness, here's the adapted sample from the above question:

import multiprocessing
from Queue import Empty

class _Popen(multiprocessing.forking.Popen):
    def __init__(self, *args, **kw):
        if hasattr(sys, 'frozen'):
            os.putenv('_MEIPASS2', sys._MEIPASS)
        try:
            super(_Popen, self).__init__(*args, **kw)
        finally:
            if hasattr(sys, 'frozen'):
                os.unsetenv('_MEIPASS2')


class Process(multiprocessing.Process):
    _Popen = _Popen


def _start():
    while True:
        try:
            command = queue.get_nowait()
        # ... and some more code to actually interpret commands
        except Empty:
            time.sleep(0.015)

def start():
    process = Process(target=_start, args=args)
    process.start()
    return process

Solution 2

To add on to nikola's answer...

*nix (Linux, Mac OS X, etc.) does NOT require any changes for PyInstaller to work. (This includes both --onedir and --onefile options.) If you only intend to support *nix systems, no need to worry about any of this.

However, if you are planning on supporting Windows, you will need to add some code, depending on which option you pick: --onedir or --onefile.

If you plan to use --onedir, all you will need to add is a special method call:

if __name__ == '__main__':
    # On Windows calling this function is necessary.
    multiprocessing.freeze_support()

According to the documentation, this call must be made immediately after if __name__ == '__main__':, or else it will not work. (It is strongly suggested that you have these two lines in your main module.)

In reality, however, you can afford to do a check before the call, and things will still work:

if __name__ == '__main__':
    if sys.platform.startswith('win'):
        # On Windows calling this function is necessary.
        multiprocessing.freeze_support()

However, calling multiprocessing.freeze_support() is possible on other platforms and situations as well - running it only impacts freezing support on Windows. If you're a bytecode nut, you'll notice that the if statement adds some bytecode, and makes potential savings from using an if statement negligible. Therefore, you should just stick to a simple multiprocessing.freeze_support() call immediately after if __name__ == '__main__':.

If you plan to use --onefile, you will need to add nikola's code:

import multiprocessing.forking
import os
import sys

class _Popen(multiprocessing.forking.Popen):
    def __init__(self, *args, **kw):
        if hasattr(sys, 'frozen'):
            # We have to set original _MEIPASS2 value from sys._MEIPASS
            # to get --onefile mode working.
            os.putenv('_MEIPASS2', sys._MEIPASS)
        try:
            super(_Popen, self).__init__(*args, **kw)
        finally:
            if hasattr(sys, 'frozen'):
                # On some platforms (e.g. AIX) 'os.unsetenv()' is not
                # available. In those cases we cannot delete the variable
                # but only set it to the empty string. The bootloader
                # can handle this case.
                if hasattr(os, 'unsetenv'):
                    os.unsetenv('_MEIPASS2')
                else:
                    os.putenv('_MEIPASS2', '')

class Process(multiprocessing.Process):
    _Popen = _Popen

# ...

if __name__ == '__main__':
    # On Windows calling this function is necessary.
    multiprocessing.freeze_support()

    # Use your new Process class instead of multiprocessing.Process

You can combine the above with the rest of his code, or the following:

class SendeventProcess(Process):
    def __init__(self, resultQueue):
        self.resultQueue = resultQueue

        multiprocessing.Process.__init__(self)
        self.start()

    def run(self):
        print 'SendeventProcess'
        self.resultQueue.put((1, 2))
        print 'SendeventProcess'

if __name__ == '__main__':
    # On Windows calling this function is necessary.
    multiprocessing.freeze_support()

    print 'main'
    resultQueue = multiprocessing.Queue()
    SendeventProcess(resultQueue)
    print 'main'

I got the code from here, PyInstaller's new site for the multiprocessing recipe. (They seem to have shut down their Trac based site.)

Note that they have a minor error with their code for --onefile multiprocessing support. They add os.sep to their _MEIPASS2 environment variable. (Line: os.putenv('_MEIPASS2', sys._MEIPASS + os.sep)) This breaks things:

  File "<string>", line 1
    sys.path.append(r"C:\Users\Albert\AppData\Local\Temp\_MEI14122\")
                                                                    ^
SyntaxError: EOL while scanning string literal

The code I provided above is the same, without the os.sep. Removing the os.sep fixes this issue and allows multiprocessing to work using the --onefile configuration.

In summary:

Enabling --onedir multiprocessing support on Windows (does NOT work with --onefile on Windows, but otherwise safe on all platforms/configurations):

if __name__ == '__main__':
    # On Windows calling this function is necessary.
    multiprocessing.freeze_support()

Enabling --onefile multiprocessing support on Windows (safe on all platforms/configurations, compatible with --onedir):

import multiprocessing.forking
import os
import sys

class _Popen(multiprocessing.forking.Popen):
    def __init__(self, *args, **kw):
        if hasattr(sys, 'frozen'):
            # We have to set original _MEIPASS2 value from sys._MEIPASS
            # to get --onefile mode working.
            os.putenv('_MEIPASS2', sys._MEIPASS)
        try:
            super(_Popen, self).__init__(*args, **kw)
        finally:
            if hasattr(sys, 'frozen'):
                # On some platforms (e.g. AIX) 'os.unsetenv()' is not
                # available. In those cases we cannot delete the variable
                # but only set it to the empty string. The bootloader
                # can handle this case.
                if hasattr(os, 'unsetenv'):
                    os.unsetenv('_MEIPASS2')
                else:
                    os.putenv('_MEIPASS2', '')

class Process(multiprocessing.Process):
    _Popen = _Popen

# ...

if __name__ == '__main__':
    # On Windows calling this function is necessary.
    multiprocessing.freeze_support()

    # Use your new Process class instead of multiprocessing.Process

Sources: PyInstaller Recipe, Python multiprocessing docs

Comments

[dano]

Does the recipe fix the issue?

[nikola]

Well, it's unclear (at least to me) how to apply the recipe. Just pasting the above code in my main Python script does not work, either, as it raises two more exceptions unrelated to my Python scripts. Which tells me that the recipe is fundamentally flawed.

[dano]

If you just run the recipe as a standalone script, does it run without errors?

[nikola]

No, I get two errors, one complaining that _MEIPASS2 is incorrectly formatted, the other that there's something wrong with pywintypes, which is incorrect.

[dano]

What version of PyInstaller?

[Daniel Pryden]

The underlying issue is that Windows doesn't support the fork() primitive the way Unix does, and multiprocessing is designed around the Unix paradigm. It works on Windows, but it works differently, and therein lies the rub. Further reading: Python multiprocessing is different under Linux and Windows; Multiprocessing on Windows breaks.

[tom10]

The ticket link is no longer valid. The current docs for this are here: github.com/pyinstaller/pyinstaller/wiki/Recipe-Multiprocessi‌​ng

[Sertalp B. Cay]

Thanks for the detailed answer. I was having issues with zombie threads appearing after I close my main Python window (with tk) when I use --onefile option. The last snippet where you redefine Popen fixed the issue. For anyone working with Python >3.4, you need to use import multiprocessing.popen_spawn_win32 as forking instead of multiprocessing.forking.

[Guido]

Don't forget that multiprocessing.freeze_support() should always be the first line in __name__ == '__main__' and that there should be no other code executed before this line (i.e. before the __name__ == '__main__'). I had some imports that executed some code, resulting in multiprocessing.freeze_support() not having any effect.

[Justin Furuness]

In 2022 on an M1 Mac, you do need to add the line for freeze_support or else it doesn't work