How to get data from inspect element of a webpage using Python

Question

I'd like to get the data from inspect element using Python. I'm able to download the source code using BeautifulSoup but now I need the text from inspect element of a webpage. I'd truly appreciate if you could advise me how to do it.

Edit: By inspect element I mean, in google chrome, right click gives us an option called inspect element which has code related to each element of that particular page. I'd like to extract that code/ just its text strings.

Solution 1

If you want to automatically fetch a web page from Python in a way that runs Javascript, you should look into Selenium. It can automatically drive a web browser (even a headless web browser such as PhantomJS, so you don't have to have a window open).

In order to get the HTML, you'll need to evaluate some javascript. Simple sample code, alter to suit:

from selenium import webdriver

driver = webdriver.PhantomJS()
driver.get("http://google.com")

# This will get the initial html - before javascript
html1 = driver.page_source

# This will get the html after on-load javascript
html2 = driver.execute_script("return document.documentElement.innerHTML;")

Note 1: If you want a specific element or elements, you actually have a couple of options -- parse the HTML in Python, or write more specific JavaScript that returns what you want.

Note 2: if you actually need specific information from Chrome's tools that is not just dynamically generated HTML, you'll need a way to hook into Chrome itself. No way around that.

Solution 2

I would like to update answer from Jason S. I wasn't able to start phantomjs on OS X

driver = webdriver.PhantomJS()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File     "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/selenium/webdriver/phantomjs/webdriver.py", line 50, in __init__
self.service.start()
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/selenium/webdriver/phantomjs/service.py", line 74, in start
raise WebDriverException("Unable to start phantomjs with ghostdriver.", e)
selenium.common.exceptions.WebDriverException: Message: Unable to start phantomjs with ghostdriver.

Resolved by answer here by downloading executables

driver = webdriver.PhantomJS("phantomjs-2.0.0-macosx/bin/phantomjs")

Solution 3

Inspect element shows all the HTML of the page which is the same as fetching the html using urllib

do something like this

import urllib
from bs4 import BeautifulSoup as BS

html = urllib.urlopen(URL).read()

soup = BS(html)

print soup.findAll(tag_name).get_text()

Comments

[user3783999]

Thank you very much. It worked perfectly except that I added the location of phantomjs.exe in the second line like this driver = webdriver.PhantomJS(executable_path=phantomjs_path)

[Ahmad Raza]

Hi, Thanks for your help. I have implemented this code using a class but does not covert javascript to html (It works ok on command line). Please help me in this regard?

[Senthil Vikram Vodapalli]

Can you update this? It doesnt work anymore in 2021

[Senthil Vikram Vodapalli]

Getting these when I ran the code. UserWarning: Selenium support for PhantomJS has been deprecated, please use headless versions of Chrome or Firefox instead. FileNotFoundError: [Errno 2] No such file or directory: 'phantomjs'