ImportError cannot import name BytesIO when import caffe on ubuntu

Question

I am trying to make caffe running on my machine equipped with Ubuntu 12.04LTS. After finishing all the steps on the Installation page, I trained the LeNet model successfully and tried to use it as the tutorial from here. Then I got the following error:

Traceback (most recent call last): 
    File "<string>", line 1, in <module>
ImportError: No module named caffe
Error in sys.excepthook:
Traceback (most recent call last):
    File "/usr/lib/python2.7/dist-packages/apport_python_hook.py", line 66, in apport_excepthook
      from apport.fileutils import likely_packaged, get_recent_crashes
    File "/usr/lib/python2.7/dist-packages/apport/__init__.py", line 1, in <module>
      from apport.report import Report
    File "/usr/lib/python2.7/dist-packages/apport/report.py", line 18, in <module>
      import problem_report
    File "/usr/lib/python2.7/dist-packages/problem_report.py", line 14, in <module>
      import zlib, base64, time, sys, gzip, struct, os
    File "/usr/lib/python2.7/gzip.py", line 10, in <module>
      import io
    File "${HOME}/path/to/caffe/python/caffe/io.py", line 2, in <module>
      import skimage.io
    File "/usr/local/lib/python2.7/dist-packages/skimage/io/__init__.py", line 11, in <module>
      from ._io import *
    File "/usr/local/lib/python2.7/dist-packages/skimage/io/_io.py", line 1, in <module>
      from io import BytesIO
ImportError: cannot import name BytesIO

Original exception was:
Traceback (most recent call last): 
    File "<string>", line 1, in <module>
ImportError: No module named caffe

I set the PYTHONPATH in .bashrc file before I did the above. What is the problem? Could anyone give some hint? I am really confused. After running the command python -c 'import io; print io.__file__'in the very directory:

Traceback (most recent call last): 
    File "${HOME}/path/to/caffe/python/caffe/io.py", line 2, in <module>
      import skimage.io
    File "/usr/local/lib/python2.7/dist-packages/skimage/io/__init__.py", line 11, in <module>
      from ._io import *
    File "/usr/local/lib/python2.7/dist-packages/skimage/io/_io.py", line 1, in <module>
      from io import BytesIO
ImportError: cannot import name BytesIO
Error in sys.excepthook:
Traceback (most recent call last):
    File "/usr/lib/python2.7/dist-packages/apport_python_hook.py", line 66, in apport_excepthook
      from apport.fileutils import likely_packaged, get_recent_crashes
    File "/usr/lib/python2.7/dist-packages/apport/__init__.py", line 1, in <module>
      from apport.report import Report
    File "/usr/lib/python2.7/dist-packages/apport/report.py", line 18, in <module>
      import problem_report
    File "/usr/lib/python2.7/dist-packages/problem_report.py", line 14, in <module>
      import zlib, base64, time, sys, gzip, struct, os
    File "/usr/lib/python2.7/gzip.py", line 10, in <module>
      import io
    File "${HOME}/path/to/caffe/python/caffe/io.py", line 2, in <module>
      import skimage.io
    File "/usr/local/lib/python2.7/dist-packages/skimage/io/__init__.py", line 11, in <module>
      from ._io import *
    File "/usr/local/lib/python2.7/dist-packages/skimage/io/_io.py", line 1, in <module>
      from io import BytesIO
ImportError: cannot import name BytesIO

Original exception was:
Traceback (most recent call last): 
    File "${HOME}/path/to/caffe/python/caffe/io.py", line 2, in <module>
      import skimage.io
    File "/usr/local/lib/python2.7/dist-packages/skimage/io/__init__.py", line 11, in <module>
      from ._io import *
    File "/usr/local/lib/python2.7/dist-packages/skimage/io/_io.py", line 1, in <module>
      from io import BytesIO
ImportError: cannot import name BytesIO

So, the problem becomes to: how to solve the name issue? P.S.: I also inserted an issue at the repository of caffe.

Solution 1

You appear to have a package or module named io in your Python path that is masking the standard library package. It is imported instead but doesn't have a BytesIO object to import.

Try running:

python -c 'import io; print io.__file__'

in the same location you are running the tutorial and rename or move the file named by that import, presuming it is not the standard library version (ending in lib/python2.7/io.pyc).

It could be you set your Python path to the wrong directory. You should include path/to/caffe/python, not path/to/caffe/python/caffe, nor should you try and run python with the latter as your current working directory. In both cases caffe/python/caffe/io.py instead of the standard library version.

The installation instructions are not at fault here; they clearly tell you to use:

export PYTHONPATH=/path/to/caffe/python:$PYTHONPATH

Note the lack of /caffe at the end of that path.

Solution 2

I ran into this problem as well, installing caffe on an AWS ubuntu 14.04 instance following the script as outlined on the BVLC github repo here: "Caffe on EC2 Ubuntu 14.04".

I have setup the python path as instructed. As diagnosed by @Martijn Pieters, the problem is that caffe is importing its own io library, which is then importing scikit-image's io library, which in turn is trying (but failing) to load the standard python io library (where BytesIO is located). Instead, due to the python path, when scikit-image tries to import BytesIO from the module io, it is circularly leading back to caffe's io module.

I also found that even when not trying to import caffe, but due to having set my python path to include caffe, that this same problem hits me elsewhere.

There are probably several ways to address this. But the essence is that the top-level import of caffe is at fault. To verify this, I altered the caffe code as follows:

1. I renamed .../caffe/io.py module to .../caffe/caffe_io.py to be safe (although with correct namespace care, this shouldn't be necessary)

2. I modified the import at the top of the pycaffe.py module from: import caffe.io to import caffe.caffe_io

3. I modified the import in __init__.py the same way (from import caffe.io to import caffe.caffe_io)

Now, when you import io from python, it won't pick up the io library in caffe. When you import caffe, it will import its custom caffe_io library, and all should be well. You may want to do a more thorough scan through the python caffe modules to ensure I haven't overlooked other places where the import needs to change.

I hope this helps. Perhaps when I have time, I'll issue a pull request with these (or similar) changes to the caffe github repo.

Comments

[stoneyang]

Hi, @Martijn Pieters, I ran the command, and get the error somewhat the same as the original post. But this time, what I am sure is that the issue rooted in the file io.py in the directory of caffe, or the file _io.py of skimage. However, I don't konw how to fix this....

[Martijn Pieters]

@stoneyang: are you running caffe/io.py as a script?

[Martijn Pieters]

@stoneyang: you say in your bug report that you set PYTHONPATH; what did you set it to? Does it include ${HOME}/path/to/caffe/python/caffe?

[Martijn Pieters]

@stoneyang: you appear to be importing caffe/io.py indeed, which suggests to me you put that directory in your PYTHONPATH; that'd be the wrong thing to do. Checking the tutorial now.

[stoneyang]

Hi, @Martijn Pieters, thanks for your effort on my problem. This is the setting PYTHONPATH=${HOME}/path/to/caffe/python/caffe:$PYTHONPATH in the original post. It was wrong. And I've tried the new settings as you suggested, and it was of no avail --- the same error received. After that, I even moved the directory out of caffe, ie to the ${HOME} directory, and redirected the PYTHONPATH to the new destination. However, still no luck :(

[Martijn Pieters]

@stoneyang: what does echo $PYTHONPATH show now? You probably extended the variable to read something like ${HOME}/path/to/caffe/python:${HOME}/path/to/caffe/python/ca‌​ffe:...., so prepended the correct path but the wrong path is still there.

[stoneyang]

@Martijin Pieters, thanks for your kindly help. I eventually didn't sovle this problem cleanly --- I just type export command every time when I need to run caffe!

[Martijn Pieters]

@stoneyang: you can also put the PYTHONPATH=... part before the python command in your shell. PYTHONPATH=... python somescript.py; it'll only apply for that one command each time.

[stoneyang]

@Martijin Pieters, that's I what I did! ;-)