ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject

Question

Importing from pyxdameraulevenshtein gives the following error, I have

pyxdameraulevenshtein==1.5.3, 
pandas==1.1.4 and 
scikit-learn==0.20.2. 
Numpy is 1.16.1. 
Works well in Python3.6, Issue in Python3.7.

Has anyone been facing similar issues with Python3.7 (3.7.9), docker image - python:3.7-buster

__init__.pxd:242: in init pyxdameraulevenshtein
    ???
E   ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject

Solution 1

I'm in Python 3.8.5. It sounds too simple to be real, but I had this same issue and all I did was reinstall numpy. Gone.

pip install --upgrade numpy

or

pip uninstall numpy
pip install numpy

Solution 2

try with numpy==1.20.0 this worked here, even though other circumstances are different (python3.8 on alpine 3.12).

Solution 3

Indeed, (building and) installing with numpy>=1.20.0 should work, as pointed out e.g. by this answer below. However, I thought some background might be interesting -- and provide also alternative solutions.

There was a change in the C API in numpy 1.20.0. In some cases, pip seems to download the latest version of numpy for the build stage, but then the program is run with the installed version of numpy. If the build version used in <1.20, but the installed version is =>1.20, this will lead to an error.

(The other way around it should not matter, because of backwards compatibility. But if one uses an installed version numpy<1.20, they did not anticipate the upcoming change.)

This leads to several possible ways to solve the problem:

• upgrade (the build version) to numpy>=1.20.0
• use minmum supported numpy version in pyproject.toml (oldest-supported-numpy)
• install with --no-binary
• install with --no-build-isolation

For a more detailed discussion of potential solutions, see https://github.com/scikit-learn-contrib/hdbscan/issues/457#issuecomment-773671043.

Solution 4

Solution without upgrading numpy

While upgrading the numpy version would often solve the issue, it's not always viable. Good example is the case when you're using tensorflow==2.6.0 which isn't compatible with the newest numpy version (it requires ~=1.19.2).

As already mentioned in FZeiser's answer, there was a change in numpys C API in version 1.20.0. There are packages that rely on this C API when they are being built, e.g. pycocotools. Given that pips dependency resolver doesn't guarantee any order for installing the packages, the following might happen:

1. pip figures out that it needs to install numpy and it chooses the latest version, 1.21.2 as of the time writing this answer.
2. It then builds a package that depends on numpy and its C API, e.g. pycocotools. This package is now compatible with numpy 1.21.2 C API.
3. At a later point pip needs to install a package that has a requirement for an older version of numpy, e.g. tensorflow==2.6.0 which would try to install numpy==1.19.5. As a result, numpy==1.21.2 is uninstalled and the older version is installed.
4. When running code that uses pycocotools, its current installation relies on the updated numpy C API, yet the numpy version was downgraded which would result in the error.

Solution

You should rebuild the package with the outdated numpy C API usage, e.g. for pycocotools:

pip uninstall pycocotools
pip install pycocotools --no-binary pycocotools

Solution 5

I had this issue when using the tensorflow object api. Tensorflow is currently NOT compatible with numpy==1.20 (although this issue is not apparent until later). In my case, the issue was caused by pycocotools. I fixed by installing an older version.

pip install pycocotools==2.0.0

View more solutions

Comments

[anymous.asker]

Didn't work for me, but upgrading to 1.20.0 did, even though the error I got was in the context of loading a library that was compiled against an earlier version of numpy.

[Ross Allen]

Hahaha...I can't believe this worked for me too! For more info, I am also in Python 3.8.5. It looks like I originally had numpy-1.19.5 and then after the uninstall-install steps I got numpy-1.20.1

[moshewe]

numpy==1.20.1 is already out with bugfixes :-)

[melaanya]

It has its own problem - when installing it removes the existing cython installation and then outputs ModuleNotFoundError: No module named 'Cython'

[A1aks]

I had to do pip install --upgrade numpy to get it upgraded.

[Danil Kononyhin]

it didn't remove cython in my case. everything worked great!

[waitingkuo]

got the same issues, fixed by upgrading numpy to 1.20.3

[wordsforthewise]

Classic unplug it and plug it back in solution. Works more often than you'd think.

[martin_wun]

Same here - uninstall and install worked for me (root cause might have been the Anaconda env on AWS using version 1.18.5 of numpy)

[Charlie Parker]

what about if you are using conda and not pip?

[mnm]

had the same issue. this solution worked for me

[Vishal Kumar Sahu]

Yup for binary incompatibility upgrading or downgrading packages is helpful.

[Smart Manoj]

Why the "OR" method?

[Ivaylo Toskov]

If the build version used in <1.20, but the installed version is =>1.20, this will lead to an error. Isn't that the other way around? You build with a newer version that doesn't have the header definition and then run it with an older version that expects the header definition. Or am I missing something?

[FZeiser]

@IvayloToskov: Although it doesn't seem intuitive, I think that it is correct: The build version is responsible for the compilation. The old build versions could not anticipate that there will be a change in numpy, so this type of setup creates a problem.

[mdev]

This was my exact problem too. In my case worked well and didn't remove Cython.

[Ivaylo Toskov]

After some verification, I can confirm that it is indeed the other way around. The change in the C API is forwards compatible, since you are removing a header. If you build with an older version that has the header and run it with a newer version, it won't break. The following sequence produces the error: pip install numpy==1.21.0, pip install pycocotools --no-binary pycocotools, pip install numpy==1.19.2, python -c "import pycocotools.coco.COCO" So the error occurs when the build version is >=1.20 and the runtime version is <1.20.

[Lewis Morris]

yup, worked for me too.

[Francesco Taioli]

Fixed with numpy==1.20.3

[Jonno Bourne]

I am in the situation you describe, however, when using pycocotools 2.0.4 the error persisted after using this method, but if I downgraded to version 2.0.0 as suggested by @royce schultz the problem resolved. Which version of pycocotools where you using?

[Ivaylo Toskov]

I don't recall exactly, but looking at the release history and the time of this answer, I think it was 2.0.2. I cannot reproduce the problem with Python 3.8 though for some reason. What is your environment? Python, pip and setuptools versions?

[Jonno Bourne]

I'm using python 3.8.10, pip 21.3.1, I didn't find setuptools or easy_tools using pip freeze, but when I ran 'pip install setuptools' it said v 58.5.2 was installed

[rdemorais]

unbelievable!! Thanks.

[Alessandro Togni]

Magic! Worked for me too!

[ibra ndiaye]

You even know what you did, you just helped me fix a problem that blocked me for 3 days