Calling multiple HTTP requests in a single HTTP request in Node.js

Question

I am trying to call multiple URL in a single URL call and push it's json response in an array and send that array in response to the end user.

My code look like this:

var express = require('express');

var main_router = express.Router();

var http = require('http');

urls = [
"http://localhost:3010/alm/build_tool",
"http://localhost:3010/alm/development_tool",
"http://localhost:3010/alm/project_architecture"];

var responses = [];

main_router.route('/')

.get(function (req, res) {

var completed_requests = 0;

for (url in urls) {

  http.get(url, function(res) {

    responses.push(res.body);

    completed_request++;

    if (completed_request == urls.length) {

        // All download done, process responses array
    }
  });
}
res.send(responses);
});

I have also tried this using npm request module. When i run this code it only return NULL or some random output that have only headers.

My aim is to call multiple URL's in a single node get request and append it's JSON output on a array and send to the end user.

Thanks

Solution 1

Here, try this code,

const async = require('async');
const request = require('request');

function httpGet(url, callback) {
  const options = {
    url :  url,
    json : true
  };
  request(options,
    function(err, res, body) {
      callback(err, body);
    }
  );
}

const urls= [
  "http://localhost:3010/alm/build_tool",
  "http://localhost:3010/alm/development_tool",
  "http://localhost:3010/alm/project_architecture"
];

async.map(urls, httpGet, function (err, res){
  if (err) return console.log(err);
  console.log(res);
});

Explanation : This code uses async and request node packages. async.map by definition takes 3 params, first one being an array, second being the iterator function you want to call with each element of that array, and the callback function, called when async.map has finished processing.

map(arr, iterator, [callback])

Produces a new array of values by mapping each value in arr through the iterator function. The iterator is called with an item from arr and a callback for when it has finished processing. Each of these callback takes 2 arguments: an error, and the transformed item from arr. If iterator passes an error to its callback, the main callback (for the map function) is immediately called with the error.

Note: All calls to iterator function are parallel.

Inside your httpGet function, you are calling request function with passed url, and explicitly telling the response format to be json. request, when finished processing, calls the callback function with three params, err - if any, res - server response, body - response body. In case there is no err from request, async.map collects the results from these callbacks as an array, and passes that array at the end to its third, callback function. Otherwise,if (err) is true, the async.map function stops the execution and calls its callback with an err.

Solution 2

I suggest to use the async library.

async.map(urls, http.get, function(err, responses){
  if (err){
    // handle error
  }
  else {
    res.send responses
  }
})

The snippet above will perform a http.get call for each of the urls in parallel, and will call your callback function with the results of all of the calls after all the responses were received.

If you want to call the urls in series, you can use async.mapSeries instead. If you want to limit the number of concurrent requests you can use async.mapLimit.

Comments

[Vaibhav Jain]

Thanks for the comment. If you are having any available code, please share. I Have tried following code but does not get anything in response. main_router.route('/') .get(function (req, res) { var test = async.each(urls, http.get, function(err, responses, callback){ if (err){ // handle error } else { return responses; } }) res.send(test); res.end(); });

[yoogeeks]

Note: async.each callback doesn't handle responses. If you need the responses at the end of call, you should use async.map which will have response of each HTTP request in the form of an array in your callback function.

[birdoftheday]

what's request here? how about async? using libraries in answers is fine but only if you specify what libraries you're using :P

[yoogeeks]

Well, asker seems to be already aware of these packages. Anyhow, updating for other users.

[Suhaib]

thanks for this. Is it possible to merge all the data bodies into one ? This is how it looks for me if I downlad JSON form multiple URls i.imgur.com/nX1IRla.png. Would you know how to merge them under 1 data body ?

[yoogeeks]

you can simply loop through each response body and add it to your final body. However, be careful if you're using pagination.