Unable to import items in scrapy

Question

I have a very basic spider, following the instructions in the getting started guide, but for some reason, trying to import my items into my spider returns an error. Spider and items code is shown below:

from scrapy.spider import BaseSpider
from scrapy.selector import HtmlXPathSelector

from myProject.items import item

class MyProject(BaseSpider):
    name = "spider"
    allowed_domains = ["website.com"]
    start_urls = [
        "website.com/start"
    ]

    def parse(self, response):
        print response.body

from scrapy.item import Item, Field

class ProjectItem(Item):
    title = Field()

When I run this code scrapy either can't find my spider, or can't import my items file. What's going on here? This should be a really example to run right?

Solution 1

I also had this several times while working with scrapy. You could add at the beginning of your Python modules this line:

from __future__ import absolute_import

More info here:

• http://www.python.org/dev/peps/pep-0328/#rationale-for-absolute-imports
• http://pythonquirks.blogspot.ru/2010/07/absolutely-relative-import.html

Solution 2

So, this was a problem that I came across the other day that I was able to fix through some trial and error, but I wasn't able to find any documentation of it so I thought I'd put this up in case anyone happens to run into the same problem I did.

This isn't so much an issue with scrapy as it is an issue with naming files and how python deals with importing modules. Basically the problem is that if you name your spider file the same thing as the project then your imports are going to break. Python will try to import from the directory closest to your current position which means it's going to try to import from the spider's directory which isn't going to work.

Basically just change the name of your spider file to something else and it'll all be up and running just fine.

Solution 3

you are importing a field ,you must import a class from items.py like from myproject.items import class_name.

Solution 4

if the structure like this:

package/
    __init__.py
    subpackage1/
        __init__.py
        moduleX.py
        moduleY.py
    subpackage2/
        __init__.py
        moduleZ.py
    moduleA.py

and if you are in moduleX.py, the way to import other modules can be:

from .moduleY.py import *

from ..moduleA.py import *

from ..subpackage2.moduleZ.py import *

refer:PEP Imports: Multi-Line and Absolute/Relative

View more solutions