Click a Button in Scrapy

Question

I'm using Scrapy to crawl a webpage. Some of the information I need only pops up when you click on a certain button (of course also appears in the HTML code after clicking).

I found out that Scrapy can handle forms (like logins) as shown here. But the problem is that there is no form to fill out, so it's not exactly what I need.

How can I simply click a button, which then shows the information I need?

Do I have to use an external library like mechanize or lxml?

Solution 1

Scrapy cannot interpret javascript.

If you absolutely must interact with the javascript on the page, you want to be using Selenium.

If using Scrapy, the solution to the problem depends on what the button is doing.

If it's just showing content that was previously hidden, you can scrape the data without a problem, it doesn't matter that it wouldn't appear in the browser, the HTML is still there.

If it's fetching the content dynamically via AJAX when the button is pressed, the best thing to do is to view the HTTP request that goes out when you press the button using a tool like Firebug. You can then just request the data directly from that URL.

Do I have to use an external library like mechanize or lxml?

If you want to interpret javascript, yes you need to use a different library, although neither of those two fit the bill. Neither of them know anything about javascript. Selenium is the way to go.

If you can give the URL of the page you're working on scraping I can take a look.

Solution 2

Selenium browser provide very nice solution. Here is an example (pip install -U selenium):

from selenium import webdriver

class northshoreSpider(Spider):
    name = 'xxx'
    allowed_domains = ['www.example.org']
    start_urls = ['https://www.example.org']

    def __init__(self):
        self.driver = webdriver.Firefox()

    def parse(self,response):
            self.driver.get('https://www.example.org/abc')

            while True:
                try:
                    next = self.driver.find_element_by_xpath('//*[@id="BTN_NEXT"]')
                    url = 'http://www.example.org/abcd'
                    yield Request(url,callback=self.parse2)
                    next.click()
                except:
                    break

            self.driver.close()

    def parse2(self,response):
        print 'you are here!'

Comments

[naeg]

It's not a real answer to my actual question, but it solves my problem nonetheless. I traced down the site request in Chrome, and found a link which shows the the information in a seperate page. Thanks! But I won't flag your answer as accepted, because others may really need to click a button.

[user]

@naeg I think the answer summarizes it correctly. You cannot click a button with Scrapy. You can send requests & receive a response. It's upto you to interpret the response with a separate javascript engine.

[naeg]

As it seems Acorn edited his answer, and now it is a full and accepted answer to my question :)

[Joseph Seung Jae Dollar]

If you yield the url after clicking the button by selenium to parse2, does it pass on the html with the content that's revealed by the click?

[Nima Soroush]

@jose I am not sure about that, but it should be easy to test

[Marius]

You can also use Scrapy-splash : blog.scrapinghub.com/2015/03/02/…