How to loop request-promise API request call?

Question

I am learning Node.JS and I am introduced to request-promise package. I use it for API call but I faced a problem where I cannot apply loop to it.

This is the example showing a simple API call:

var read_match_id = {
    uri: 'https://api.steampowered.com/IDOTA2Match_570/GetMatchHistory/V001',
    qs: {
        match_id: "123",
        key: 'XXXXXXXX'
    },
    json: true
};

rp(read_match_id)
.then(function (htmlString) {
    // Process html...
})
.catch(function (err) {
    // Crawling failed...
});

How can i have loop like this:

 var match_details[];
 for (i = 0; i < 5; i++) {
     var read_match_details = {
              uri: 'https://api.steampowered.com/IDOTA2Match_570/GetMatchDetails/V001',
            qs: {
                  key: 'XXXXXXXXX',
                  match_id: match_id[i]
                },
            json: true // Automatically parses the JSON string in the response 
    };
    rp(read_match_details)
       .then (function(read_match){
            match_details.push(read_match)//push every result to the array
        }).catch(function(err) {
            console.log('error');
        });
    }

And how can I know when all the async request are done?

Solution

request-promise uses Bluebird for Promise.

The simple solution is Promise.all(ps), where ps is array of promises.

var ps = [];
for (var i = 0; i < 5; i++) {
    var read_match_details = {
        uri: 'https://api.steampowered.com/IDOTA2Match_570/GetMatchDetails/V001',
        qs: {
            key: 'XXXXXXXXX',
            match_id: match_id[i]
        },
        json: true // Automatically parses the JSON string in the response 
    };
    ps.push(rp(read_match_details));
}

Promise.all(ps)
    .then((results) => {
        console.log(results); // Result of all resolve as an array
    }).catch(err => console.log(err));  // First rejected promise

The only disadvantage of this is, this will go to catch block immediately after any of the promise is rejected. 4/5 resolved, won't matter, 1 rejected will throw it all to catch.

Alternative approach is to use Bluebird's inspection (refer this). We'll map all promises to their reflection, we can do an if/else analysis for each promise, and it will work even if any of the promise is rejected.

// After loop
ps = ps.map((promise) => promise.reflect()); 

Promise.all(ps)
    .each(pInspection => {
        if (pInspection.isFulfilled()) {
            match_details.push(pInspection.value())
        } else {
            console.log(pInspection.reason());
        }
    })
    .then(() => callback(match_details)); // Or however you want to proceed

Hope this will solve your problem.

Comments

[brendan]

thanks for your reply.. i think i will take time to digest since i am just a beginner to it.. I feel like asking whether is this the best approach to handle async request? or this is just the complicated one

[Ashwani Agarwal]

TBH, I've just started NodeJS few months back. Usually for such situations, Bulk APIs are used, where you can get all such data, quickly with a single request. But since these APIs are public, I don't think there will be any bulk API available. There's nothing wrong with this approach, (other users will point out otherwise). Just one thing, don't just copy paste, go through the definition of each function before using it. Async nature of node, is a little different to grasp, but once you understand it, your jaws will be dropped, and you'll unlock the full potential of NodeJS.

[parsley72]

@AshwaniAgarwal what's the rp() function in your first example?

[Ashwani Agarwal]

@parsley72 instance of request-promise library