cx_freeze - including my own modules?

Question

I have a small application built with PyQt4 that I'm trying to freeze with cx_freeze, but I'm running into an issue with cx_freeze including some of my own modules that are required for the application to work.

I have two modules that are imported in my application that are located in a folder above where the application is located. I.e.:

Application path:

Python\DataViewer-PyQt4\DataViewer.py

Other modules:

Python\My Analysis Packages\Ephystools

Python\My Analysis Packages\PrairieAnalysis

In my application I import these by using (if they're not in my python path already)

sys.path.append(os.path.abspath('../My Analysis Packages'))

I have tried including PrairieAnalysis and EphysTools in both 'includes' and 'packages' in my setup.py file. I have tried including 'My Analysis Packages' as well. I have tried providing the paths to these as well.

They all contain init.py files, as the actual application is capable of importing them just fine.

If I put PrairieAnalysis and/or EphysTools in the 'includes' list then setup.py build returns an ImportError:

 File "C:\Anaconda3\lib\site-packages\cx_Freeze\finder.py", line 386, in _ImportModule
    raise ImportError("No module named %r" % name)
ImportError: No module named 'PrairieAnalysis'

If I leave them out of 'includes' setup.py build completes, but then when I go to open the application I get that same error.

I've looked through the various cx_freeze module import questions but none seem to have dealt with this particular scenario.

My actual setup.py:

# -*- coding: utf-8 -*-

import sys
from cx_Freeze import setup, Executable

base = None
if sys.platform == 'win32':
    base = 'Win32GUI'

options = {
    'build_exe': {
        'includes': ['atexit', 'PrairieAnalysis', 'EphysTools'],
    }
}

executables = [
    Executable('DataViewer.py', base=base)
]

setup(name='DataViewer',
      version='0.1',
      description='Application for viewing Prairie-generated csv data files',
      options=options,
      executables=executables
      )

Edit 1: Output from os.getcwd() in setup.py file:

D:\OneDrive\Documents\Python\DataViewer-PyQt4

Output from sys.path in setup.py file:

    ['D:\\OneDrive\\Documents\\Python\\DataViewer-PyQt4', 'D:\\OneDrive\\Documents\\Python\\My Analysis Packages', 'C:\\Anac
    onda3\\python34.zip', 'C:\\Anaconda3\\DLLs', 'C:\\Anaconda3\\lib', 'C:\\Anaconda3', 'C:\\Anaconda3\\lib\\site-packages',
     'C:\\Anaconda3\\lib\\site-packages\\Sphinx-1.2.3-py3.4.egg', 'C:\\Anaconda3\\lib\\site-packages\\win32', 'C:\\Anaconda3
    \\lib\\site-packages\\win32\\lib', 'C:\\Anaconda3\\lib\\site-packages\\Pythonwin', 'C:\\Anaconda3\\lib\\site-packages\\r
    unipy-0.1.1-py3.4.egg', 'C:\\Anaconda3\\lib\\site-packages\\setuptools-7.0-py3.4.egg']

Edit 2:

So I've also tried using py2exe and I run into the same issue. If I include the packages in "includes" I get the following traceback:

Traceback (most recent call last):
  File "setup.py", line 7, in <module>
    setup(windows=['DataViewer.py'], options={"py2exe": {"includes" :["sip", "PyQt4.QtCore", "PyQt4.QtGui", "PrairieAnal
ysis", "EphysTools"]}})
  File "C:\Anaconda3\lib\distutils\core.py", line 148, in setup
    dist.run_commands()
  File "C:\Anaconda3\lib\distutils\dist.py", line 955, in run_commands
    self.run_command(cmd)
  File "C:\Anaconda3\lib\distutils\dist.py", line 974, in run_command
    cmd_obj.run()
  File "C:\Anaconda3\lib\site-packages\py2exe\distutils_buildexe.py", line 188, in run
    self._run()
  File "C:\Anaconda3\lib\site-packages\py2exe\distutils_buildexe.py", line 267, in _run
    builder.analyze()
  File "C:\Anaconda3\lib\site-packages\py2exe\runtime.py", line 164, in analyze
    mf.import_hook(modname)
  File "C:\Anaconda3\lib\site-packages\py2exe\mf3.py", line 120, in import_hook
    module = self._gcd_import(name)
  File "C:\Anaconda3\lib\site-packages\py2exe\mf3.py", line 274, in _gcd_import
    return self._find_and_load(name)
  File "C:\Anaconda3\lib\site-packages\py2exe\mf3.py", line 318, in _find_and_load
    loader = importlib.find_loader(name, path)
  File "C:\Anaconda3\lib\importlib\__init__.py", line 87, in find_loader
    name=name)
ImportError: namespace packages do not have loaders

In this case I put my two packages (PrairieAnalysis and EphysTools) into my site-packages folder. Why are my packages being treated differently than the other packages?

Edit 3: So I have gotten py2exe to work by using the following setup script:

from distutils.core import setup
import py2exe
import PrairieAnalysis.pv_import
import EphysTools.utilities

includes = ["sip", "PyQt4.QtCore", "PyQt4.QtGui", "PrairieAnalysis", "EphysTools", "lxml._elementpath"]
packages = ["PrairieAnalysis", "EphysTools"]

setup(windows=['DataViewer.py'], options={"py2exe": {"includes" : includes,
                                                     "packages": packages}})

just importing PrairieAnalysis and EphysTools alone didn't work though, nor did doing

from PrairieAnalysis import *
from EphysTools import *

Adding those import statements to my cx_freeze setup.py script, however, does not fix the issue.

Edit 4:

>>> import PrairieAnalysis
>>> print(PrairieAnalysis.__file__)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute '__file__'
>>> print(PrairieAnalysis.__init__)
<method-wrapper '__init__' of module object at 0x0000000002B9C9F8>

Edit 5:

>>> os.listdir('D:\\OneDrive\\Documents\\Python\\My Analysis Packages')
['.idea', 'EphysTools', 'PrairieAnalysis', '__init___.py']
>>> os.listdir('D:\\OneDrive\\Documents\\Python\\My Analysis Packages\\PrairieAnalysis')
['misc_code.py', 'pv_import.py', 'pxml_parse.py', '__init___.py', '__pycache__']
>>> os.listdir('D:\\OneDrive\\Documents\\Python\\My Analysis Packages\\EphysTools')
['synaptics.py', 'utilities.py', '__init___.py', '__pycache__']

Solution

You need to modify sys.path in your setup.py script in the same way that you do in your application. cx_Freeze looks at sys.path to find the modules and packages to include in your build, so if the directory containing those packages is not on sys.path, it can't find them.

Edit: It turned out that the problem was a misnamed __init__.py file. The package was still importable as a PEP 420 namespace package, but cx_Freeze doesn't handle those yet.

Comments

[dan_g]

That was my first thought, but doing that did not solve the problem.

[dan_g]

And in fact, the sys.path line is actually in a try..except block since on the computer that I do most of my coding I actually have the 'My Analysis Packages' included by default for my python path since i use the modules so often. The sys.path.append line is only included in the application in case I'm working on one of my other computers where I haven't made that change to ensure that those modules are always available for import

[Thomas K]

Can you get your setup.py script to print os.getcwd() and sys.path before calling setup(), and check that it's looking where you think it's looking for those packages?

[dan_g]

Done, see my edit in the question for what those return. The correct path to where PrairieAnalysis and EphysTools are contained is in sys.path. I have also tried just puttin PrairieAnalysis and EphysTools into my site-packages folder. Doing this I can import them just fine from anywhere on my machine (terminal, another script, whatever), and yet they still do not get copied over / I get an ImportError. Really at a loss as to what is going on here. Is there something besides init that needs to be in the module folder?

[dan_g]

I can even put import PrairieAnalysis and import EphysTools in the actual setup.py file. This does not produce an ImportError (setup.py runs if I do not put either of those in includes, but neither is copied over and running DataViewer.exe then gives the traceback discussed above)

[Thomas K]

No, __init__.py should do the trick. So, just to check, D:\OneDrive\Documents\Python\My Analysis Packages\PrairieAnalysis\__init__.py exists, yes? Is that what you get if you start Python and do import PrairieAnalysis; print(PrairieAnalysis.__file__)?

[dan_g]

Yes, D:\OneDrive\Documents\Python\My Analysis Packages\PrairieAnalysis_init_.py exists. print(PrairieAnalysis.__file__) returns a traceback, though. print(PrairieAnalysis.__init__) works however. See Edit 4 for more details.

[Thomas K]

That sounds like it's found it as a namespace package for some reason. Could you also show the output of os.listdir('D:\\OneDrive\\Documents\\Python\\My Analysis Packages\\PrairieAnalysis')?

[dan_g]

See Edit 5. What seems odd is setup.py is generating the compiled python files (hence the inclusion of pycache), but they aren't being copied over properly.

[Thomas K]

Gah, I see it. You have __init___.py, with three underscores after the word init. Rename that to __init__.py.

[dan_g]

oy, that's both embarrassing and infuriating. Thank you. What was the third underscore doing? Since I could still import the module within the application itself?

[Thomas K]

Folders without an __init__.py are now importable as namespace packages (see PEP 420). cx_Freeze has not been updated to handle those yet.

[Alaa M.]

This didn't work for me... cx_Freeze still can't find my module. I don't understand though, you didn't write the module name under packages. Is that intentional?

[ZLNK]

Yes, I made that intentional, the other way it didn't work. The other alternative can be use datadir to locate the module, as is shows here: cx-freeze.readthedocs.io/en/latest/faq.html