Installing Python module with pip

Question

I'm trying to install a module called Scrapy. I installed it using

pip install Scrapy

I see the 'scrapy' folder in my /usr/local/lib/python2.7/site-packages, but when I try to import it in a Python program, is says there is no module by that name. Any ideas as to why this might be happening?

EDIT: Here is the output of the pip command:

    Downloading/unpacking Scrapy
  Downloading Scrapy-0.20.0.tar.gz (745kB): 745kB downloaded
  Running setup.py egg_info for package Scrapy

    no previously-included directories found matching 'docs/build'
Requirement already satisfied (use --upgrade to upgrade): Twisted>=10.0.0 in /usr/local/lib/python2.7/site-packages (from Scrapy)
Requirement already satisfied (use --upgrade to upgrade): w3lib>=1.2 in /usr/local/lib/python2.7/site-packages (from Scrapy)
Requirement already satisfied (use --upgrade to upgrade): queuelib in /usr/local/lib/python2.7/site-packages (from Scrapy)
Requirement already satisfied (use --upgrade to upgrade): lxml in /usr/local/lib/python2.7/site-packages (from Scrapy)
Requirement already satisfied (use --upgrade to upgrade): pyOpenSSL in /usr/local/lib/python2.7/site-packages (from Scrapy)
Requirement already satisfied (use --upgrade to upgrade): cssselect>=0.9 in /usr/local/lib/python2.7/site-packages (from Scrapy)
Requirement already satisfied (use --upgrade to upgrade): zope.interface>=3.6.0 in /usr/local/lib/python2.7/site-packages (from Twisted>=10.0.0->Scrapy)
Requirement already satisfied (use --upgrade to upgrade): six>=1.4.1 in /usr/local/lib/python2.7/site-packages (from w3lib>=1.2->Scrapy)
Requirement already satisfied (use --upgrade to upgrade): setuptools in /usr/local/lib/python2.7/site-packages/setuptools-1.1.6-py2.7.egg (from zope.interface>=3.6.0->Twisted>=10.0.0->Scrapy)
Installing collected packages: Scrapy
  Running setup.py install for Scrapy
    changing mode of build/scripts-2.7/scrapy from 644 to 755

    no previously-included directories found matching 'docs/build'
    changing mode of /usr/local/bin/scrapy to 755
Successfully installed Scrapy
Cleaning up...

When I run /usr/local/bin/scrapy I get the usage for the command and the available commands. I noticed that I have a python2.7 and python2.7-32 in my /usr/local/bin, and I remember installing the 32 bit version because of a problem with Mavericks.

Here is the output of python /usr/local/bin/scrapy:

Traceback (most recent call last): File "/usr/local/bin/scrapy", line 3, in <module> from scrapy.cmdline import execute ImportError: No module named scrapy.cmdline 

And head /usr/local/bin/scrapy:

#!/usr/local/opt/python/bin/python2.7 from scrapy.cmdline import execute execute()

Solution 1

Are you using Homebrew or MacPorts or something? As @J.F.Sebastian said, it sounds like you are having issues mixing the default python that comes with OS X, and one that is installed via a package manager... Try /usr/local/opt/python/bin/python2.7 -m scrapy and see if that throws an ImportError.

If that works, then you may want to consider making that python executable your default. Something like alias python2.7=/usr/local/opt/python/bin/python2.7 and then always use python2.7 instead of the default python. You can likewise just point python to the /urs/local... bit, but then you won't have easy access to the system (OS X-supplied) python if you ever needed it for some reason.

Solution 2

EDIT: You can force pip to install to an alternate location. The details are here: Install a Python package into a different directory using pip?. If you do indeed have extra Python folders on your system, maybe you can try directing scrapy to those, even if just for a temporary solution.

Can you post the output of the pip command? Perhaps it is failing somewhere?

Also, is it possible you have two versions of Python on your machine? Pip only installs to one location, but perhaps the version of Python on your path is different.

Finally, sometimes package names given to pip are not exactly the same as the name used to import. Check the documentation of the package. I took a quick look and the import should be lowercase:

import scrapy

Solution 3

When all else fails you can always set the environment variable PYTHONPATH (see Permanently add a directory to PYTHONPATH for help) to the path where you installed Scrapy. (pending you're not using virtualenv -- and if you are please specify so we can help, it's generally a good idea to provide OS too)

Comments

[jfs]

is there /usr/local path in sys.path?

[jfs]

@Miles: If /path/to/scrapy shows usage info but python /path/to/scrapy raises ImportError then there are at least two python executables on your machine and pip doesn't install into the default one that is available as python. What head /usr/local/bin/scrapy shows?

[jfs]

@Jud: "even though the package itself is not installed there" is incorrect. scrapy is installed into /usr/local/opt/... otherwise scrapy command would produce ImportError instead of the usage info. @cm2's answer suggests a simple way to access particular python executable. If python2.7 alias is defined to point to #!/usr/local/opt/python/bin/python2.7 then it is enough to add #!/usr/bin/env python2.7 shebang to any Python script that has import scrapy in it or just start python2.7 from the command-line.

[Mike]

I added the output of the pip command to my post. As far as I'm aware I only have python 2.7.5. I also tried import scrapy to no avail.

[Mike]

It turns out that the version of Python I had installed via homebrew was conflicting with the Python that I manually installed.

[cm2]

Glad you got it figure out. I think all of us were saying very similar things, RE making sure scrapy is using the right python.