Fetch vs Request

Question

I'm consuming a JSON stream and am trying to use fetch to consume it. The stream emits some data every few seconds. Using fetch to consume the stream gives me access to the data only when the stream closes server side. For example:

var target; // the url.
var options = {
  method: "POST",
  body: bodyString,
} 
var drain = function(response) {
  // hit only when the stream is killed server side.
  // response.body is always undefined. Can't use the reader it provides.
  return response.text(); // or response.json();
};
var listenStream = fetch(target, options).then(drain).then(console.log).catch(console.log);

/*
    returns a data to the console log with a 200 code only when the server stream has been killed.
*/

However, there have been several chunks of data already sent to the client.

Using a node inspired method in the browser like this works every single time an event is sent:

var request = require('request');
var JSONStream = require('JSONStream');
var es = require('event-stream');

request(options)
.pipe(JSONStream.parse('*'))
.pipe(es.map(function(message) { // Pipe catches each fully formed message.
      console.log(message)
 }));

What am I missing? My instinct tells me that fetch should be able to mimic the pipe or stream functionality.

Solution

response.body gives you access to the response as a stream. To read a stream:

fetch(url).then(response => {
  const reader = response.body.getReader();

  reader.read().then(function process(result) {
    if (result.done) return;
    console.log(`Received a ${result.value.length} byte chunk of data`);
    return reader.read().then(process);
  }).then(() => {
    console.log('All done!');
  });
});

Here's a working example of the above.

Fetch streams are more memory-efficient than XHR, as the full response doesn't buffer in memory, and result.value is a Uint8Array making it way more useful for binary data. If you want text, you can use TextDecoder:

fetch(url).then(response => {
  const reader = response.body.getReader();
  const decoder = new TextDecoder();

  reader.read().then(function process(result) {
    if (result.done) return;
    const text = decoder.decode(result.value, {stream: true});
    console.log(text);
    return reader.read().then(process);
  }).then(() => {
    console.log('All done!');
  });
});

Here's a working example of the above.

Soon TextDecoder will become a transform stream, allowing you to do response.body.pipeThrough(new TextDecoder()), which is much simpler and allows the browser to optimise.

As for your JSON case, streaming JSON parsers can be a little big and complicated. If you're in control of the data source, consider a format that's chunks of JSON separated by newlines. This is really easy to parse, and leans on the browser's JSON parser for most of the work. Here's a working demo, the benefits can be seen at slower connection speeds.

I've also written an intro to web streams, which includes their use from within a service worker. You may also be interested in a fun hack that uses JavaScript template literals to create streaming templates.

Comments

[sparkFinder]

thanks for the answer! As mentioned in code comments I'm consistently seeing response.body as undefined against the same example where the XHR version works. Any idea why that might be? We do control the data source and are making each new object on a new line for exactly the reasons you outlined. @jaffa-the-cake

[sparkFinder]

Any suggestions on the body issue? stackoverflow.com/users/123395/jaffa-the-cake

[JaffaTheCake]

response.body being undefined suggests the browser hasn't implemented streams yet. What browser & version are you using?

[Shane Gannon]

XHR approaches (such as request.js) buffer the full response in memory? e.g. If I http GET a 1 GB file it'll consume 1 GB of RAM before I stream it to disk?