Node.js HTTP response streams

Question

Using the native http.get() in Node.js, I'm trying to pipe a HTTP response to a stream that I can bind data and end events to.

I'm currently handling this for gzip data, using:

http.get(url, function(res) {
  if (res.headers['content-encoding'] == 'gzip') {
    res.pipe(gunzip);
    gunzip.on('data', dataCallback);
    gunzip.on('end', endCallback);
  }
});

Gunzip is a stream and this just works. I've tried to create streams (write streams, then read streams) and pipe the response, but haven't been having much luck. Any suggestions to replicate this same deal, for non-gzipped content?

Solution 1

The response object from a HTTP request is an instance of readable stream. Therefore, you would collect the data with the data event, then use it when the end event fires.

var http = require('http');
var body = '';

http.get(url, function(res) {
  res.on('data', function(chunk) {
    body += chunk;
  });
  res.on('end', function() {
    // all data has been downloaded
  });
});

The readable.pipe(dest) would basically do the same thing, if body in the example above were a writable stream.

Solution 2

Nowadays the recommended way of piping is using the pipeline function. It is supposed to protect you from memory leaks.

const { createReadStream} = require('fs');
const { pipeline } = require('stream')
const { createServer, get } = require('http')

const errorHandler = (err) => err && console.log(err.message);

const server = createServer((_, response) => {
  pipeline(createReadStream(__filename), response, errorHandler)
  response.writeHead(200);
}).listen(8080);

get('http://localhost:8080', (response) => {
  pipeline(response, process.stdout, errorHandler);
  response.on('close', () => server.close())
});

Another way of doing it that has more control would be to use async iterator

async function handler(response){
  let body = ''
  for await (const chunk of response) {
    let text = chunk.toString()
    console.log(text)
    body += text
  }
  console.log(body.length)
  server.close()
}

get('http://localhost:8080', (response) => handler(response).catch(console.warn));

Comments

[dzm]

I'm trying to get the response body as it comes, then once it's finished. I thought I could do res.on('data') but that never seems to trigger.

[dzm]

Doh, well it turned out to be due to http not following redirects. Your code is what I have and it works fine, with the exception of redirects. I checked out npm follow-redirects and will test it out.

[vkurchatkin]

You can also try popular request module, which also follows redirects

[dzm]

@vkurchatkin request does support gzipped content

[Jwan622]

Why do you have to do string concatenation? is chunk not the entirety of the data?

[hexacyanide]

You need to do string concatenation because the response is a stream. The data event fires when there's data immediately available, which may or may not be the entire response.

[Ati Ranzuglia]

Looks like there is a little mistake, from NodeJs guide: Remember, the request object is a ReadableStream and the response object is a WritableStream.

[Ati Ranzuglia]

Disregard my comment, I assumed this was the server side.