How do I read the contents of a Node.js stream into a string variable?

Question

I'm hacking on a Node program that uses smtp-protocol to capture SMTP emails and act on the mail data. The library provides the mail data as a stream, and I don't know how to get that into a string.

I'm currently writing it to stdout with stream.pipe(process.stdout, { end: false }), but as I said, I need the stream data in a string instead, which I can use once the stream has ended.

How do I collect all the data from a Node.js stream into a string?

Solution 1

(This answer is from years ago, when it was the best answer. There is now a better answer below this. I haven't kept up with node.js, and I cannot delete this answer because it is marked "correct on this question". If you are thinking of down clicking, what do you want me to do?)

The key is to use the data and end events of a Readable Stream. Listen to these events:

stream.on('data', (chunk) => { ... });
stream.on('end', () => { ... });

When you receive the data event, add the new chunk of data to a Buffer created to collect the data.

When you receive the end event, convert the completed Buffer into a string, if necessary. Then do what you need to do with it.

Solution 2

Another way would be to convert the stream to a promise (refer to the example below) and use then (or await) to assign the resolved value to a variable.

function streamToString (stream) {
  const chunks = [];
  return new Promise((resolve, reject) => {
    stream.on('data', (chunk) => chunks.push(Buffer.from(chunk)));
    stream.on('error', (err) => reject(err));
    stream.on('end', () => resolve(Buffer.concat(chunks).toString('utf8')));
  })
}

const result = await streamToString(stream)

Solution 3

None of the above worked for me. I needed to use the Buffer object:

  const chunks = [];

  readStream.on("data", function (chunk) {
    chunks.push(chunk);
  });

  // Send the buffer or you can put it into a var
  readStream.on("end", function () {
    res.send(Buffer.concat(chunks));
  });

Solution 4

Hope this is more useful than the above answer:

var string = '';
stream.on('data',function(data){
  string += data.toString();
  console.log('stream data ' + part);
});

stream.on('end',function(){
  console.log('final output ' + string);
});

Note that string concatenation is not the most efficient way to collect the string parts, but it is used for simplicity (and perhaps your code does not care about efficiency).

Also, this code may produce unpredictable failures for non-ASCII text (it assumes that every character fits in a byte), but perhaps you do not care about that, either.

Solution 5

What do you think about this ?

async function streamToString(stream) {
    // lets have a ReadableStream as a stream variable
    const chunks = [];

    for await (const chunk of stream) {
        chunks.push(Buffer.from(chunk));
    }

    return Buffer.concat(chunks).toString("utf-8");
}

View more solutions

Comments

[19h]

You should copy the stream or flag it with (autoClose: false). It is bad practice to pollute the memory.

[arcseldon]

A couple of lines of code illustrating the answer is preferable to just pointing a link at the API. Don't disagree with the answer, just don't believe it is complete enough.

[sean2078]

What would be a more efficient way to collect string parts? TY

[Tom Carchrae]

you could use a buffer docs.nodejitsu.com/articles/advanced/buffers/how-to-use-buff‌​ers but it really depends on your use.

[Valeriu Paloş]

Use an array of strings where you append each new chunk to the array and call join("") on the array at the end.

[Nicolas Le Thierry d'Ennequin]

Useful answer but it looks like each chunk must be converted to a string before it is pushed in the array: chunks.push(chunk.toString());

[alextgordon]

This isn't right. If buffer is halfway through a multi-byte code point then the toString() will receive malformed utf-8 and you will end up with a bunch of � in your string.

[Tom Carchrae]

i'm not sure what you mean by a 'mutli-byte code point' but if you want to convert the encoding of the stream you can pass an encoding parameter like this toString('utf8') - but by default string encoding is utf8 so i suspect that your stream may not be utf8 @alextgordon - see stackoverflow.com/questions/12121775/… for more

[Ivo]

this is actually the cleanest way of doing it ;)

[Simon A. Eugster]

With newer node.js versions, this is cleaner: stackoverflow.com/a/35530615/271961

[Bryan Johnson]

Works great. Just a note: if you want a proper string type, you will need to call .toString() on the resulting Buffer object from concat() call

[Dan Dascalescu]

The answer should be updated to not recommend using a Promises library, but use native Promises.

[ControlAltDel]

@DanDascalescu I agree with you. The problem is that I wrote this answer 7 years ago, and I haven't kept up with node.js . If you are someone else would like to update it, that would be great. Or I could simply delete it, as there seems to be a better answer already. What would you recommend?

[Dan Dascalescu]

@ControlAltDel: I appreciate your initiative to delete an answer that is no longer the best. Wish others had similar discipline.

[ControlAltDel]

@DanDascalescu Unfortunately, when I tried to delete, SO won't let me as it is the selected answer.

[TOPKAT]

This is the only one that worked for me ! Great thanks

[JohnK]

I'm really new to streams and promises and I'm getting this error: SyntaxError: await is only valid in async function. What am I doing wrong?

[Enclo Creations]

You have to call the streamtostring function within a async function. To avoid this you can also do streamToString(stream).then(function(response){//Do whatever you want with response});

[MultiplyByZer0]

This should be the top answer. Congratulations on producing the only solution that gets everything right, with (1) storing the chunks as Buffers and only calling .toString("utf8") at the end, to avoid the problem of a decode failure if a chunk is split in the middle of a multibyte character; (2) actual error handling; (3) putting the code in a function, so it can be reused, not copy-pasted; (4) using Promises so the function can be await-ed on; (5) small code that doesn't drag in a million dependencies, unlike certain npm libraries; (6) ES6 syntax and modern best practices.

[J-D3V]

This was a great answer!

[Mike Yermolayev]

@alextgordon is right. In some very rare cases when I had a lot of chunks I got those � at the start and end of chunks. Especially when there where russian symbols on the edges. So it's correct to concat chunks and convert them on end instead of converting chunks and concatenating them. In my case the request was made from one service to another with request.js with default encoding

[Krisztián Balla]

Why not move the chunks array into the promise?

[Andrei LED]

I have noticed that above code could fail with Uncaught TypeError [ERR_INVALID_ARG_TYPE]: The "list[0]" argument must be an instance of Buffer or Uint8Array. Received type string if the stream produces string chunks instead of Buffer. Using chunks.push(Buffer.from(chunk)) should work with both string and Buffer chunks.

[Andrei LED]

After I came up with essentially the same code using current top answer as the hint, I have noticed that above code could fail with Uncaught TypeError [ERR_INVALID_ARG_TYPE]: The "list[0]" argument must be an instance of Buffer or Uint8Array. Received type string if the stream produces string chunks instead of Buffer. Using chunks.push(Buffer.from(chunk)) should work with both string and Buffer chunks.

[ViRuSTriNiTy]

Works, very clean, no dependencies, nice!

[zevero]

What a nice and unexpected solution! You are a Mozart of javascript streams and buffers!

[Rafael Beckel]

Turns out the actual best answer came late to the party: stackoverflow.com/a/63361543/1677656

[Rafael Beckel]

Turns out the actual best answer came late to the party: stackoverflow.com/a/63361543/1677656

[Rafael Beckel]

Turns out the actual best answer came late to the party: stackoverflow.com/a/63361543/1677656

[Jan]

Had to use chunks.push(Buffer.from(chunk)); to make it work with string chunks.

[blub]

chunks.push(Buffer.from(chunk)) gives a typescript error Argument type Buffer | string is not assignable to parameter type WithImplicitCoercion<ArrayBuffer | SharedArrayBuffer>. Is this a valid error or are the types broken?

[ban_javascript]

What's the reason for this edit? I write my code more similarly to the previous revision (directly passing reject as the callback instead of calling it)--will that run into errors?

[ban_javascript]

Wow, this looks very neat! Does this have any problems (aside from the one mentioned in the above comment)? Can it handle errors?

[Marlon Bernardes]

@ban_javascript The reasoning behind the edit is that is generally considered bad practice to pass a callback to a function that was not designed for it. There was nothing wrong with the previous version, though - it should work just fine. One could argue it might break if the Promise.resolve API ever changes, but that's very unlikely to happen. I've just edited it in order to encourage what I see as best practices.

[Epic Speedy]

This is the modern equivalent to the top answer. Whew Node.js/JS changes fast. I'd recommend using this one as opposed to the top rated one as it's much cleaner and doesn't make the user have to touch events.

[Dirk Schumacher]

My IDE claims 'for await (let...' to be errornous.

[Armen Michaeli]

@DirkSchumacher Your IDE either uses outdated script interpreter (for await is a valid ECMAScript syntax) or is itself outdated if it attempts to (unsuccessfully) execute some code containing for await. Which IDE is it? Anyway, IDEs aren't designed to actually run programs "in production", they lint them and help with analysis during development.

[Dirk Schumacher]

@amn I am running in the most up to date IntelliJ-Ultimate. Though it is more a Java IDE it knows somewhat web stuff as well. I am sitting behind a company firewall so this and other things may be the issue. Thanks for your advise/info/idea!

[Armen Michaeli]

@DirkSchumacher No bother. Just see if you can find out exactly what component of your IDE -- I assume it will be a program -- loads and executes the script containing for await. Query the version of the program and find out if the version actually supports the syntax. Then find out why your IDE is using the particular "outdated" version of the program and find a way to update both.

[thescientist]

FYI, had to push the chunk using Buffer.from, ex. chunks.push(Buffer.from(chunk)), but otherwise a nice and elegant solution! thanks!

[Marius Mircea]

If you attempt to read a big file (text) and do something with the data, u need to append the next chunk based on prev chunk (remaining invalid bytes) to ensure that in the case a chunk cuts off your bytes, you are re-computing them on the next read. To do that use string_decoder. Here's a usage example Edit: I'd advise streaming it line-by-line if you need to work with the file data: readLine.createInterface({ input: stream, crlfDelay: Infinity })

[SaidAkh]

this is the only way to do it correctly

[Philippe]

In my case I was happy with const packagePath = execSync("rospack find my_package").toString().