Installing package dependencies for Scrapy

Question

So among the many packages users need to install for Scrapy, I think I'm having trouble with pyOpenSSL.

When I try to get a tutorial Scrapy project created, I get this following output:

Traceback (most recent call last):
  File "C:\Python27\lib\runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "C:\Python27\lib\runpy.py", line 72, in _run_code
    exec code in run_globals
  File "C:\Python27\lib\site-packages\scrapy\cmdline.py", line 168, in <module>
    execute()
  File "C:\Python27\lib\site-packages\scrapy\cmdline.py", line 122, in execute
    cmds = _get_commands_dict(settings, inproject)
  File "C:\Python27\lib\site-packages\scrapy\cmdline.py", line 46, in _get_comma
nds_dict
    cmds = _get_commands_from_module('scrapy.commands', inproject)
  File "C:\Python27\lib\site-packages\scrapy\cmdline.py", line 29, in _get_comma
nds_from_module
    for cmd in _iter_command_classes(module):
  File "C:\Python27\lib\site-packages\scrapy\cmdline.py", line 20, in _iter_comm
and_classes
    for module in walk_modules(module_name):
  File "C:\Python27\lib\site-packages\scrapy\utils\misc.py", line 68, in walk_mo
dules
    submod = import_module(fullpath)
  File "C:\Python27\lib\importlib\__init__.py", line 37, in import_module
    __import__(name)
  File "C:\Python27\lib\site-packages\scrapy\commands\bench.py", line 3, in <mod
ule>
    from scrapy.tests.mockserver import MockServer
  File "C:\Python27\lib\site-packages\scrapy\tests\mockserver.py", line 6, in <m
odule>
    from twisted.internet import reactor, defer, ssl
  File "C:\Python27\lib\site-packages\twisted\internet\ssl.py", line 59, in <mod
ule>
    from OpenSSL import SSL
  File "build\bdist.win32\egg\OpenSSL\__init__.py", line 8, in <module>
  File "build\bdist.win32\egg\OpenSSL\rand.py", line 11, in <module>
  File "build\bdist.win32\egg\OpenSSL\_util.py", line 3, in <module>
ImportError: No module named cryptography.hazmat.bindings.openssl.binding

And when I googled that last error (no module named cryptography.hazmat), I see a couple of mentions of pyOpenSSL. So I went ahead and tried running easy_install pyOpenSSL==0.14 to make sure it's the latest version, but when I do that, I get this output:

c:\python27\include\pymath.h(22) : warning C4273: 'round' : inconsistent dll lin
kage
        C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\INCLUDE\math.h(51
6) : see previous definition of 'round'
c:\users\bk\appdata\local\temp\easy_install-tztawu\cryptography-0.4\temp\easy_in
stall-svxsjy\cffi-0.8.2\c\misc_win32.h(225) : error C2632: 'char' followed by 'b
ool' is illegal
c:\users\bk\appdata\local\temp\easy_install-tztawu\cryptography-0.4\temp\easy_in
stall-svxsjy\cffi-0.8.2\c\misc_win32.h(225) : warning C4091: 'typedef ' : ignore
d on left of 'unsigned char' when no variable is declared
c/_cffi_backend.c(5295) : warning C4146: unary minus operator applied to unsigne
d type, result still unsigned
c/_cffi_backend.c(5296) : warning C4146: unary minus operator applied to unsigne
d type, result still unsigned
c/_cffi_backend.c(5297) : warning C4146: unary minus operator applied to unsigne
d type, result still unsigned
c/_cffi_backend.c(5298) : warning C4146: unary minus operator applied to unsigne
d type, result still unsigned
error: Setup script exited with error: command '"C:\Program Files (x86)\Microsof
t Visual Studio 12.0\VC\BIN\cl.exe"' failed with exit status 2

So I'm a bit lost as to what I need to do to get Scrapy up and running properly

Solution 1

I got the same error on Mac OS.

I solved it by using openssl 0.13 instead of the latest version.

easy_install pyOpenSSL==0.13

or

pip install pyOpenSSL==0.13

Solution 2

I highly recommend using conda instead of pip, especially when using Windows. Among many other things it will grab the appropriate binary files for your system. It makes setting up a scientific python environment (think Scipy, Numpy, Pandas ...) a breeze.

So, read up on Anaconda, install Anaconda, then do:

conda create -n scrapyenv python=2  # creates a new py2 environment
activate scrapyenv                  # switch to the new environment
conda install scrapy                # install scrapy

Steps one and two are only necessary if you want it encapsulated in a seperate environment. Btw, a whole battery of useful packages will be installed if you do conda install anaconda.

Additionaly, in case conda does not include pyOpenSSL or you do NOT want to install anaconda please take a look at point 9 of the tutorial How to install Scrapy in 64bit Windows 7.