How to create streams from string in Node.Js?

Question

I am using a library, ya-csv, that expects either a file or a stream as input, but I have a string.

How do I convert that string into a stream in Node?

Solution 1

From node 10.17, stream.Readable have a from method to easily create streams from any iterable (which includes array literals):

const { Readable } = require("stream")

const readable = Readable.from(["input string"])

readable.on("data", (chunk) => {
  console.log(chunk) // will be called once with `"input string"`
})

Note that at least between 10.17 and 12.3, a string is itself a iterable, so Readable.from("input string") will work, but emit one event per character. Readable.from(["input string"]) will emit one event per item in the array (in this case, one item).

Also note that in later nodes (probably 12.3, since the documentation says the function was changed then), it is no longer necessary to wrap the string in an array.

https://nodejs.org/api/stream.html#stream_stream_readable_from_iterable_options

Solution 2

As @substack corrected me in #node, the new streams API in Node v10 makes this easier:

const Readable = require('stream').Readable;
const s = new Readable();
s._read = () => {}; // redundant? see update below
s.push('your text here');
s.push(null);

… after which you can freely pipe it or otherwise pass it to your intended consumer.

It's not as clean as the resumer one-liner, but it does avoid the extra dependency.

(Update: in v0.10.26 through v9.2.1 so far, a call to push directly from the REPL prompt will crash with a not implemented exception if you didn't set _read. It won't crash inside a function or a script. If inconsistency makes you nervous, include the noop.)

Solution 3

Do not use Jo Liss's resumer answer. It will work in most cases, but in my case it lost me a good 4 or 5 hours bug finding. There is no need for third party modules to do this.

NEW ANSWER:

var Readable = require('stream').Readable

var s = new Readable()
s.push('beep')    // the string you want
s.push(null)      // indicates end-of-file basically - the end of the stream

This should be a fully compliant Readable stream. See here for more info on how to use streams properly.

OLD ANSWER: Just use the native PassThrough stream:

var stream = require("stream")
var a = new stream.PassThrough()
a.write("your string")
a.end()

a.pipe(process.stdout) // piping will work as normal
/*stream.on('data', function(x) {
   // using the 'data' event works too
   console.log('data '+x)
})*/
/*setTimeout(function() {
   // you can even pipe after the scheduler has had time to do other things
   a.pipe(process.stdout) 
},100)*/

a.on('end', function() {
    console.log('ended') // the end event will be called properly
})

Note that the 'close' event is not emitted (which is not required by the stream interfaces).

Solution 4

Just create a new instance of the stream module and customize it according to your needs:

var Stream = require('stream');
var stream = new Stream();

stream.pipe = function(dest) {
  dest.write('your string');
  return dest;
};

stream.pipe(process.stdout); // in this case the terminal, change to ya-csv

or

var Stream = require('stream');
var stream = new Stream();

stream.on('data', function(data) {
  process.stdout.write(data); // change process.stdout to ya-csv
});

stream.emit('data', 'this is my string');

Solution 5

Edit: Garth's answer is probably better.

My old answer text is preserved below.

---

To convert a string to a stream, you can use a paused through stream:

through().pause().queue('your string').end()

Example:

var through = require('through')

// Create a paused stream and buffer some data into it:
var stream = through().pause().queue('your string').end()

// Pass stream around:
callback(null, stream)

// Now that a consumer has attached, remember to resume the stream:
stream.resume()

View more solutions

Comments

[pathikrit]

I get this: TypeError: string is not a function at String.CALL_NON_FUNCTION (native) when I use it like new StringStream(str).send()

[mpen]

I couldn't get zeMirco's solution to work for my use case, but resumer worked quite well. Thanks!

[Garth Kidd]

The @substack resumer suggestion worked very well for me. Thanks!

[Jolly Roger]

Resumer is great, but the "auto-resumes the stream on nextTick" can yield surprises if you expect you can pass the stream to unknown consumers! I had some code that piped a content stream to a file if a db save of metadata succeeded. That was a lurking bug, it happened to succeed when the db write returned success immediately! I later refactored things to be inside an async block, and boom, the stream was never readable. Lesson: if you don't know who's going to consume your stream, stick to the through().pause().queue('string').end() technique.

[Felix Rabe]

From the docs (link): "All Readable stream implementations must provide a _read method to fetch data from the underlying resource."

[b3s1m0t7]

This code breaks stream conventions. pipe() is supposed to return the destination stream, at very least.

[B T]

The end event isn't called if you use this code. This is not a good way to create a stream that can be used generally.

[Sukima]

Just because JavaScript uses duck typing doesn't mean you should reinvent the wheel. Node already provides an implementation for streams. Just create a new instance of stream.Readable like @Garth Kidd suggested.

[icktoofay]

@Sukima: stream.Readable did not exist when I wrote this answer.

[B T]

I spent about 5 hours debugging my code because I used the resumer part of this answer. It'd be great if you could like.. remove it

[Jim Jones]

@eye_mew you need to require('stream') first

[sdc]

@GarthKidd I am trying to use this example with a readline interface. example In my case I need to use the 'line' event and 'close' event. The 'line' event is triggering, but I can't seem to get the 'close' event to fire. Any ideas?

[sdc]

If anyone comes across the same issue that I had, I found a solution. From inside the 'line' event I had to manually call rl.close() when I came across a given criteria rl.on('line', (line) => { if(line.indexOf('\n\n') > -1) { rl.close(); } });

[dopatraman]

Why do you push null into the stream's buffer?

[dopatraman]

Also, how would I stream an Array, given that a string is just an Array of characters?

[chrishiestand]

@dopatraman null tells the stream that it has finished reading all the data and to close the stream

[Chris Allen Lane]

When I posted this initially, I did not include the relevant code, which I was told is frowned upon.

[Kirill Reznikov]

what is the purpose of the return at the end?

[Philippe T.]

"always return something (or nothing)" , this exemple from the documentation .

[Kirill Reznikov]

In JS, if a function doesn't have a return it is an equivalent to your empty return. Could you please provide a link where you have found it?

[Philippe T.]

you should right . I said that more for best practice. I want to return nothing , it 's not a mistake . So i remove the line.

[masterxilo]

Is this a pun on "there's an app for that"? ;)

[Axel Rauschmayer]

Looks like you shouldn’t do it this way. Quoting the docs: “The readable.push() method is intended be called only by Readable Implementers, and only from within the readable._read() method.”

[B T]

@Finn You don't need the parens in javascript if there aren't any args

[Dem Pilafian]

The link in the comment is the useful one: npmjs.com/package/string-to-stream

[Trenton D. Adams]

According to the nodejs docs the example is violating proper usage. i.e. those s.push() calls should be inside of the _read implementation. See the second last comment in the docs nodejs.org/api/stream.html#stream_readable_push_chunk_encodi‌​ng

[stackdave]

dont' use "var" in 2018! but const

[Russell Briggs]

FYI I tried to use this library for writing JSON to google drive but it wouldn't work for me. Wrote an article about this here: medium.com/@dupski/… . Also added as an answer below

[abbr]

According to stream.Readable.from, Calling Readable.from(string) or Readable.from(buffer) will not have the strings or buffers be iterated to match the other streams semantics for performance reasons.

[Fizker]

My bad. The function was added in 10.7, and behaved the way I originally described. Sometime since, strings no longer need to be wrapped in arrays (since 12.3, it no longer iterates each character individually).

[electrovir]

@TrentonD.Adams it appears that that comment is from the v10.x docs. The v14.x docs simply say "The readable.push() method is used to push the content into the internal buffer" which implies to me that it's a perfectly fine method to use.

[cjol]

These solutions explain various ways to create a stream, but none of them the question, which is asking how to convert a string into a stream.

[raarts]

Maybe, but it still helped me to fix my own (similar) problem.

[Zikoat]

eslint is complaining about the Unexpected empty arrow function. I replaced this with image._read = () => undefined; to satisfy it.