How to fetch data over multiple pages?

Question

My project is based on React, redux, redux-saga, es6 and I try to fetch data from this API:

http://api.dhsprogram.com/rest/dhs/data/BD,2000,2004,2007?&returnFields=CharacteristicLabel,Indicator,IndicatorId,Value&f=json

As you can see, this specific API call shows data with a limit of 100 data per page spread on 40 pages.

According to this answer: http://userforum.dhsprogram.com/index.php?t=msg&th=2086&goto=9591&S=Google it says that you can extend the limit to a maximum of 3000data per page.

However, in some cases I would do an API call that exceeds that limit which means I would not receive all my data doing it like this:

export function fetchMetaData(countryCode: string, surveyYears: string) {
return (fetch('http://api.dhsprogram.com/rest/dhs/data/' + countryCode + ',' + surveyYears + '?returnFields=CharacteristicLabel,Indicator,IndicatorId,Value&f=json')
    .then(response => response.json())
    .then(json => json.Data.map(survey => survey)))
} 

So my question is; what is the best way to get all data from this API given that I know the total pages of data. The answer in the forum link suggest to loop through the API. However, I can't find the right syntax usage to do this.

My idea would be doing one api call to get the total number of pages. Then store this in a state using redux+redux-saga. Then do a new request sending the total pages as parameter and fetch this total number of pages times. And by doing this I can't figure out the syntax to store the data for each iteration.

Solution 1

A possible solution -the idea is to get the number of pages first, then make the appropriate number of API calls, pushing the promise from each call into an array. We then wait for all the promises to resolve, and do something with the returned data.

async function fetchMetaData() {

    const response = await fetch('apiUrlToGetPageNumber');

    const responses = await Promise.all(
        Array.from(
            Array(resp.data.pagesRequired),
            (_, i) => fetch(`apiUrlToSpecificPage?page=${i}`)
        )
    );
    
    // do something with processedResponses here

}
            
            

Solution 2

Here is another possible solution using async/await. The beauty of this is that the total_pages count is dynamic, so that if it increases while you're processing your request, it'll make sure you get it all.

async function fetchMetaData() {
  let allData = [];
  let morePagesAvailable = true;
  let currentPage = 0;

  while(morePagesAvailable) {
    currentPage++;
    const response = await fetch(`http://api.dhsprogram.com/rest/dhs/data?page=${currentPage}`)
    let { data, total_pages } = await response.json();
    data.forEach(e => allData.unshift(e));
    morePagesAvailable = currentPage < total_pages;
  }

  return allData;
}

Comments

[Kien Vu]

Thank you very much! (I upvoted answer but I have less then 15rep). However, by using .reduce on processedResponses; the function can' recall that prev is defined. But I get the idea, so I will try to figure it out!

[Alex Young]

Good point on the reduce, I will edit into something that works.

[Kien Vu]

I would appreciate that =)

[Kien Vu]

I have tried to make this works ever since. I just learned that Javascript is not running each line/statement sequential (which is new to me). In my case "processedResponses" are supposed to get all the resolved data, but since every statement is racing for the finish line my return value is an empty array. I wonder if you (or anyone) know a solution to this? I am currently researching generators in ES6 in hope that may be the solution. But I can't get it to work yet.

[hcdocs]

What would this look like with a required Rest-User-Token?

[vitalbone]

@hcdocs you'd need to pass in options into the fetch call with the headers included. It'd look something like: fetch('http://api.dhs...', { headers: { "Rest-User-Token": "your-token" } }). Docs for supplying request options to fetch