Scipy - Sparse Library ImportError: DLL load failed: %1 is not a valid Win32 application

Question

I have recently been moving my programming to a 64-bit windows 7 machine and have been install the relevant libraries. However I am having issues with the Scipy - Sparse library.

I have installed scipy0.12.0-amd64-py27 (as my python 2.7 install is the 64-bit release) release for windows and when using the scipy library directly I encounter no errors. e.g.

import scipy
print scipy.version

returns

<module 'scipy.version' from 'C:\Python27\lib\site-packages\scipy\version.pyc'>

As expected.

However when trying the import the sparse library as follows:

from scipy import sparse

I get:

ImportError: DLL load failed: %1 is not a valid Win32 application.

Sadly my knowledge is limited and I cannot seem to find a relevant solution. The closest was that I had installed the wrong library, however I cannot see why this would be the case if the root scipy library was working correctly.

Does anyone know of a solution to this? Or would it be better to use one of the python packages for windows with scipy installed directly?

Thanks

edit:

Here is the traceback from the error:

Traceback (most recent call last):
  File "C:\Users\Piers\Documents\Physics\PhD\Code\Far Time Crank-Nicholson\FarTimeCrankNicholson.py", line 16, in <module>
    from scipy import sparse
  File "C:\Python27\lib\site-packages\scipy\sparse\__init__.py", line 182, in <module>
    from .csr import *
  File "C:\Python27\lib\site-packages\scipy\sparse\csr.py", line 15, in <module>
    from .sparsetools import csr_tocsc, csr_tobsr, csr_count_blocks, \
  File "C:\Python27\lib\site-packages\scipy\sparse\sparsetools\__init__.py", line 5, in <module>
    from .csr import *
  File "C:\Python27\lib\site-packages\scipy\sparse\sparsetools\csr.py", line 26, in <module>
    _csr = swig_import_helper()
  File "C:\Python27\lib\site-packages\scipy\sparse\sparsetools\csr.py", line 22, in swig_import_helper
    _mod = imp.load_module('_csr', fp, pathname, description)
ImportError: DLL load failed: %1 is not a valid Win32 application.

Solution

Ultimately, this means that scipy.sparse itself, or something it imports, either is, or depends on at load time, a .DLL or .pyd file that's broken or for the wrong-architecture.

So, there are two steps to tracking this down.

---

First, you need to figure out which actual .pyd/.DLL file is raising this exception. Unfortunately, Python 2.7 will not give you this information directly.

You may be able to figure it out by looking at the traceback from the ImportError—it should be something imported by the lowest module in the chain. (If you don't understand the traceback, paste it into your answer, and hopefully someone else can tell you.)

Failing that, you will have to walk through things manually. You can look at the source to scipy/sparse/__init__.py in your site-packages or online in the repository to see what it does, then try each import one by one. When you find the one that fails, look at its source code, and so on, until you reach the bottom of the tree.

Now that you've pasted the traceback, we can tell from this part:

File "C:\Python27\lib\site-packages\scipy\sparse\sparsetools\csr.py", line 22, in swig_import_helper
  _mod = imp.load_module('_csr', fp, pathname, description)

… that the problem is that scipy.sparse.csr, aka scipy/sparse/csr.py is failing to load scipy.sparse._csr, which is almost certainly a file named something like _csr.dll or _csr.pyd, and somewhere inside the scipy directory. Exactly where may be a bit complicated, because the code is calling imp.load_module with a custom pathname instead of using a simple import statement… but finding it shouldn't be too hard. (If you get stuck, you can always temporarily edit csr.py to print the pathname before passing it to load_module.)

So, that's the library that you need to check.

---

Now that you the .pyd/.dll file actually responsible, there are two possibilities.

First, it may itself be broken, or for the wrong architecture (e.g., 32-bit instead of 64-bit), etc.

If not, if must link against another .dll that's broken or for the wrong architecture.

The tool Dependency Walker can tell you whether a DLL is 32- or 64-bit, whether it's corrupted too badly to load, etc. And it can also give you a list of all the DLLs that the library itself depends on, which you can then check until you find the problem.

---

Meanwhile, if you're wondering how you got into this situation in the first place…

Installing binary Python packages on Windows generally does not install the dependencies they need, or detect that they're missing—especially non-Python dependencies. So, you don't see the problem until you try to use some part of the code that requires those dependencies. Christoph Gohkle's does a great job trying to minimize these problems (generally just installing the appropriate MSVC redistributable, and the Numpy-MKL version of numpy, is all you need), but there are limits.

As the scipy website explains, it's a lot easier to download a "scientific Python distribution" like Anaconda or Enthought.

Or, alternatively, if you install from source—e.g., via pip—you will have to have all of the prerequisites properly set up, or the installation will fail early and tell you what's wrong. (Sometimes it's a little cryptic—e.g., if you're using the gcc toolchain but didn't install gfortran, the error message just tells you something about not being about to compile config.f… but at least it's something you can search for, or post on SO and get an immediate answer for.)

Comments

[Piers Lillystone]

Thanks for the quick reply. I have attached the traceback here. Normally I am ok at following tracebacks, but this one stumped me as the final line wasn't an import. I don't believe it is the .csr import. However several lines earlier in the code is an "import imp" statement, which could be the offending line of code. Would you agree?

[abarnert]

@PiersLillystone: It's not the import imp; it's the imp.load_module('_csr', …). That's a complicated way of doing the equivalent of import _csr, but overriding various things to specify a pathname instead of letting Python find it normally.

[Piers Lillystone]

Funny you should say that because it seems like this is a python 2.6.0+ feature. If the version is below 2.6.0 the code will excute the import _csr.

[abarnert]

@PiersLillystone: Then it sounds like it would have been a bit easier to track down the problem (or at least the first half of it) in Python 2.5… but ultimately, it's going to be the same problem. Most likely, either _csr.pyd, or something it depends on, is a 32-bit DLL (or possibly a corrupted file), and you have to track down which one it is and figure out how to get the right one instead.

[Piers Lillystone]

Thanks for all the help. I've been running some tests using scipy.test() to see what else is wrong. Seems to be several dlls are missing. Probably going to do a fresh install of python 2.7 and install numpy/scipy first to avoid conflicts (I already have several non-related libraries installed, which could complicate debugging). Really appreciate the help though.