How to Find The Link for JSON Data of a Certain Website

Question

I just finished a tutorial on how to develop an android application that retrieves updated posts from a Blog using JSON data.

The link to the JSON data used to retrieve the posts, was the blog name ending with "/api/get_recent_summary"

How can I find the link of the JSON data for different websites?

For example the website for The Time Magazine http://time.com

Solution 1

The quickest and easiest way is to use google developer tools in Google Chrome.

1st go to google developer tools. F12 or Ctrl+Shift+I or top right Ellipsis->More Tools->Developer Tools

2nd Click on the "Network" tab.

3rd click on the "XHR" sub-tab. XHR(XMLHttpRequest)

if a site uses json it will be listed under the XHR sub-tab. You can search through the different return objects, select one and use the "preview" sub-sub-tab to view it.

View JSON

View JSON URL

Although the above way is the easiest it is not the most stable way of getting the info you need. Many sites make changes to the return data without any notice. This will probably break your app...

I think what you are looking for is an API(Application programming interface). Most web APIs return json or xml. You should start by searching for the api documentation for the specific site that you want to get json data from. Example documentation for sites that have public api feeds are github api or youtub api. Many of these will require authentication in order to get the desired json response but the documentation should show you how to do it.

Using a legitimate web api is the most stable way to go. Meaning your app has less chance of randomly breaking all of the time due to feed changes, url changes... I hope this helps!

Solution 2

I know this is an older question, but I felt compelled to chime in. If you goal is to simply determine if a site uses JSON for data exchange, then the solution proposed by Tim is a very good solution. However, if you are looking to scrape data from an arbitrary site, there is no guarantee that the site uses JSON as data exchange, as stated by @KenanZahirovic. There are numerous sites that do not do this. Instead they embed the data into the HTML, or use XML, or some other method for getting the content to the client. There is no standard way of doing it, which is why data scraping is so difficult. It requires figuring out how data is transferred and building a client for that.

For scenarios where you need to gather data from multiple sources, you may end up with multiple clients due to the nuances between the sources. This site explains some best practices for data scraping. However, this would likely require a server-side application. Having a server-side application that gathers data and stores it in a database would make the most sense for this scenario. This way you can have a consistent API that the client hits to access the data.

An algorithm has been proposed that can scrape many sites fairly confidently. If you only want to have a client, this may or may not be the best way to go. It all depends on how much processing you want the client to do. If at all possible, try and off load processing to a server.

This answer might be way more than is required, but ,again, I felt compelled to chime in. I am sure the previous answer was sufficient. I do recommend though that you mark an answer as the accepted answer.

Best of luck!

Comments

[Kenan Zahirovic]

There is no such thing as mandatory JSON source of data for every website. Still, there are some techniques - more info at en.wikipedia.org/wiki/Web_scraping

[AMS91]

@KenanZahirovic, So how can I know of a website is offering a JSON source of data?

[Kenan Zahirovic]

AFAIK, You can't be sure. There is no universal solution.

[S. W. G.]

this partially answer my same question, however once determined it is using JSON is there any way to download the JSON archive / text file used?

[Tim]

@S.Redrum Are you asking how to find the link of the JSON file or how to actually download the contents of the link?

[S. W. G.]

I think it gives the same result, what I need is to obtain the full archive of the JSON either data or file works for me. In my case I could not find a .json file since there is no request URL but rather a POST to a web address.

[Tim]

@S.Redrum So you already have a URL? The URL does not need to end in .json. I know both android Java and Kotlin both have ways of doing REST calls. There are plenty of examples on how to do this. how to post in Android There are lots of other examples. If you are using a different programming language a simple search would return a lot of answers...

[S. W. G.]

I see that the page POST to another page, this one does the XHR call that is intercepted by firefox/chrome and load the same page plus the data looked up showed in a grid. What should I do to access the whole JSON data and/or download it?

[S. W. G.]

Hello, I just need to download the full content of the json archive, be it in form of a text file or other formats in case.