Python scraping pdf from URL

Question

I want to scrape the text from the URL "http://www.nycgo.com/venues/thalia-restaurant#menu" The text I'm interested in is in the 'menu' tab on the page. I tried BeautifulSoup to get all the text on the page, but the return value from the following code misses all the text in the menu.

html = urllib2.urlopen("http://www.nycgo.com/venues/thalia-restaurant#menu")
html=html.read()
soup = BS(html)
print soup.get_text()

It seems that the content of the menu is part of the html on the page when I inspect elements from the menu content. I did notice that when physically browsing the page, it takes several seconds for the menu to fully load. Not sure if that's why the code above fails to get the menu content.

Any insight would be appreciated.

Solution

While soup.get_text() will return all of the text from a HTML document (webpage) the problem here is that the menu is embedded in the page as a PDF, which Beautiful soup cannot access. The actual PDF file is defined in Javascript like follows:

{
    name: "menu",
    show: Boolean(1),
    url: "/assets/files/programs/rw/2016W/thalia-restaurant.pdf"
}

The simplest way to extract this then is probably to use regular expressions. While this is generally a bad idea, here you're looking for a very specific thing — a file, wrapped in "quotes" ending in .pdf. The following code will find that and extract the URL:

import re
from urllib import urlopen

html = urlopen("http://www.nycgo.com/venues/thalia-restaurant#menu")
html_doc = html.read()

match = re.search(b'\"(.*?\.pdf)\"', html_doc)
pdf_url = "http://www.nycgo.com" + match.group(1).decode('utf8')

Now pdf_url is:

u'http://www.nycgo.com/assets/files/programs/rw/2016W/thalia-restaurant.pdf'

However, extracting the text from the PDF is a little trickier. You can download the file first:

from urllib import urlretrieve
urlretrieve(pdf_url, "download.pdf")

Then extract the text as described using the function in this answer to another question:

text = convert_pdf_to_txt("download.pdf")
print(text)

Returns:

NEW YOUR CITY 
RESTAURANT WEEK

WINTER 2016

MONDAY - FRIDAY
828 Eighth Avenue
New York City, 10019

Tel: [phone]

www.restaurantthalia.com

LUNCH $25
FIRST COURSE
CREAMY POLENTA
fricassee of truffle mushrooms

...

Comments

[amphetamachine]

If there isn't any special reason this has to be done using a Python script, I'd suggest using wkhtmltopdf.

[jumbopap]

The content for the page is dynamically loaded with Javascript. You're not going to be able to get all the content simply by downloading the HTML text.

[Camuslu]

@jumbopap thanks, i had the suspicion that something like that could be the reason the return value misses the menu content. Any suggestion how to deal with this?

[Camuslu]

@amphetamachine thanks, I tried the tool but the pdf created still misses the menu content :(