Pyinstaller error ImportError: No module named 'requests.packages.chardet.sys

Question

I can't seem to find the root cause of this. I don't know if it's pyinstaller, a pip problem, the requests module, or something else as nothing can be eliminated conclusively.

I wrote a script in python that properly configures a new hardware sonicwall for our enterprise network when we have to deploy a new unit. It configures a proper .exp file in memory, logs into the sonicwall device with default credentials, imports the file via a multi-part data form, restarts the sonicwall, then logs in again and changes the shared secret properly. For security reasons, I can't post the code here, but I can explain the problem with a much simpler example. Previously, the code was using urllib and urllib2 to process http requests, but then I discovered the requests module when I had to re-write the script to include csrfTokens.

Long story short, the script works amazing when called by the python interpreter. However, when trying to compile it with pyinstaller, I get a series of errors now that I've switched to requests instead of the urllibs.

Some more background:

Windows 7 - Python2.7.9
pip 6.0.8 from C:\Python27\lib\site-packages\pip-6.0.8-py2.7.egg (python 2.7)
pip freeze output:
pyinstaller==2.1.1.dev0
pywin32==219
requests==2.5.3

As an example, I'll give you some code that bombs out, a simple request to google.

#!/usr/bin/python
import requests 
r = requests.get('https://google.com') 
print(r.text)

The above code works for a simple request when I call the file from python, but when I compile it, I get this:

(EDIT Having problems pasting output, here is the pastebin) pastebin

It creates a windows executable, but this is the following error when I try to run it:

NOTE I did install the ms c++ 2008 redistributable to help clear up the msvcr90.dll, but I still get the above requests.packages.chardet.sys error.

I've tried everything I can think of, installing chardet, installing chardet2, installing cchardet, forcing earlier versions of pyinstaller and requests incrementally. Scrapping pip and manually installing pyinstaler and requests. I'm at my wits end with things to try, I don't quite get the error on hand here. requests.packages.chardet exists on the system. I also have C:\Python27 in my windows PATH as I can call python from any directory.

If you need more information, please let me know. I tried to be as thorough with the errors as possible and what I have installed, but I can provide more if needed.

ALSO This issue ONLY appears to be happening when I try to compile when I import requests. creating test scripts, beautifulsoup, urllib/2, etc all compile a valid windows exe that runs properly.

Solution 1

I don't have a solution for this yet, but this is caused by latest changes in requests module (versions 2.5.2 & 2.5.3).

For now you can use version 2.5.1 until PyInstaller will have suitable hook for solving this issue.

I cannot really explain the issue, but it looks like there's some kind of collision between PyInstaller import hooks and some latest additions to requests (VendorAlias).

Solution 2

Good news, this has been fixed in the latest version of requests

pip install requests --upgrade

Easy.

Solution 3

As m1keil says, the problem is between PyInstaller import hooks and the new load_module features of requests, implemented in the requests.packages.__init__.py source file.

Debugging this file you can see that any import of the requests package is passing through the load_module function. This includes the python standard packages. This the reason of the error.

My workaround consists in editing the requests.packages.__init__.py (inside the virtualenv folder) of the version 2.5.3 and adding this piece of code at the very beginning of the load_module function:

    print "Requested name = ", name #Comment this line when it works
    direct_loaded_packages = ('sys', 'errno','logging','warnings'
            ,'socket','os','re','time','hashlib','base64'
            ,'time','collections','datetime','io', 'argparse'
            ,'codecs', 'Queue', 'zlib', 'ssl', 'operator'
            ,'types','platform','struct', 'StringIO','httplib'
            ,'simplejson','cookielib','urllib','urlparse'
            ,'urllib2','Cookie','http','binascii','certifi'
            ,'uuid','json','threading','dummy_thread','email'
            ,'email.utils','operator','mimetypes')    


    new_name = ''
    #Package with three directory deep 
    if '.'.join(name.split('.')[3:]) in direct_loaded_packages:
        new_name = '.'.join(name.split('.')[3:])
    #Package with four directory deep
    elif '.'.join(name.split('.')[4:]) in direct_loaded_packages:
        new_name = '.'.join(name.split('.')[4:])
    #Package with five directory deep
    elif '.'.join(name.split('.')[5:]) in direct_loaded_packages:
        new_name = '.'.join(name.split('.')[5:])

    if new_name != '':
        module = __import__(new_name)
        return module 

By test&error and grepping I have build the list of packages that must be directly loaded without passing by the load_module function.

Isn't elegant but it works. I've tested a more elegant solution but I haven't succeeded.

Comments

[swdev]

... Thanks! You just answered this yesterday, and I got the error today. Awesome!

[Richard Cascarina]

m1keil, you just saved what's left of the hair on my head! I've been pulling my hair out over this issue and forcing requests 2.5.1 resolved it! Before I posted here, I also posted on the requests github issue tracker, so he is aware of it. I linked this thread to his issue tracker as well. But just so I understand, we don't think it's necessarily all requests fault, or pyinstaller, just seems like both are doing things differently and running into a collision with import hooks and a new addition to requests?

[m1keil]

Yes, that's exactly what I think. The issue with things such as PyInstaller, py2exe and such is that the underlying Python module needs to be "aware" that sometimes its running in "frozen" state. It looks like this is exactly the case.

[swdev]

Just wandering, is the case and the solution will be the same with twisted import error on importing zope.interface? * I am not sure, I think it's something call fake import in PyInstaller

[m1keil]

@swdev, personally I don't know. I didn't have time to get too much details about the error and I'm not familiar with zope.