Scraping dynamic content in a website
Solution 1
The polite option would be to ask the owners of the site if they have an API which allows you access to their news stories.
The less polite option would be to trace the HTTP transactions that take place while the page is loading and work out which one is the AJAX call which pulls in the data.
Looks like it's this one. But it looks like it might contain session data, so I don't know how long it will continue to work for.
Solution 2
If the content is generated dynamically, you can use Windmill
or Seleninum
to drive the browser and get the data once it's been rendered.
You can find an example here.
Aks
Updated on June 04, 2022Comments
-
Aks almost 2 years
I need to scrape news announcements from this website, Link. The announcements seem to be generated dynamically. They dont appear in the source. I usually use mechanize but I assume it wouldnt work. What can I do for this? I'm ok with python or perl.
-
Aks over 12 yearsHow did you identify the file?
-
Øyvind Skaar over 12 yearsProbably firebug or the new web-console in firefox 8.. or similar.
-
Dave Cross over 12 yearsI'm old-skool. I used the HTTP Live Headers extension for Firefox.
-
Aks over 12 years@ØyvindSkaar I have firebug, could you tell me how to look for it?
-
Øyvind Skaar over 12 years@Aks Don't have it installed, but there's "Net" tab or something that can show you all the GET (and POST etc) requests the browser does when loading the page. This includes the ones made by the javascript code. Those probably appear far down. As you use the site, you can see what ajax calls (just GET's, POST's etc) are made. See getfirebug.com/network
-
choroba over 12 yearsDoes this handle javascript and similar client-side stuff?
-
Aks over 12 yearsok. thanks. btw, will the _t parameter change or can I use it as a link?
-
Adam Morris over 12 yearsNope. You're better off with jcollado's answer, looking at Windmill or Selenium if you're on python. Haven't used them though.
-
Dave Cross over 12 yearsNo-one can possibly know what the _t parameter is. As I implied above, it might be some kind of session information. I suppose you might get a clue if you examined the Javascript that makes the request. Or, the best solution is probably to ask the site owners to explain the parameters to their AJAX API. You really shouldn't just use it without talking to them first.