How to convert a Javascript Object to a Node Buffer?

Question

I'm using Buffer on my node server and Buffer on my Javacript client.

For the purposes of saving bytes, I'm looking to send my data to the server through websockets as binary as opposed to JSON.

So, if I had the Javascript object of [ 5, false, 55, "asdf" ], I'd like to convert that to the buffer on the client right before sending it. Maybe something like this:

object.toBuffer('int16', 'bool', 'int16', 'utf8');

and read it on the server something like this:

var obj = buffer.read('int16', 'bool', 'int16', 'utf8');

I'm looking at current solutions and it looks like I may have to just do a lot of concating, specifying byte offsets/lengths, converting from ints to booleans, etc.

Is there a better way?

Edit: Here's how I think you currently have to do it. I guess my issue is just that it's too verbose and error-prone and I'm looking for a more concise and elegant way to do it, because this operation will be performed in many different places in my code.

// On client for [ 5, false, 55, "test" ]

const writeFirst = Buffer.allocUnsafe(2);
writeFirst.writeInt16LE(5, 0);
const writeSecond = Buffer.allocUnsafe(1);
writeSecond.writeUInt8(0);
const writeThird = Buffer.allocUnsafe(2);
writeThird.writeInt16LE(55, 0);
const writeFourth = Buffer.from('test');

const result = Buffer.concat([writeFirst, writeSecond, writeThird, writeFourth]);

// On server for reading buffer of [ 5, false, 55, "test" ]

const readFirst = result.readInt16LE(0);
const readSecond = Boolean(result.readUInt8(2));
const readThird = result.readInt16LE(3);
const readFourth = result.toString('utf8', 5);

Edit #2: Was googling around and I think I might want something like protocol buffers. I'm not exactly sure what they are yet or if they apply but it looks like you can specify a schema in a file for all your messages and then serialize your JSON objects to that schema and have it return a buffer, which you can then deserialize using the same schema on the client/other server. I'm going to look in to this some more.

Solution 1

A buffer's first argument must be a: String, Buffer, ArrayBuffer, Array, or array-like object.

Taking that information into account, we could implement what you are looking for by creating a buffer from a String. It would look something like the following:

let json = [ 5, false, 55, 'asdf' ];

let buffer = Buffer.from(JSON.stringify(json));
console.log('Buffer: ', buffer); // Buffer: <Buffer 5b 20 35 2c 20 66 61 6c 73 65 2c 20 35 35 2c 20 22 61 73 64 66 22 20 5d>

Then you can bring your JSON back like so:

let converted = JSON.parse(buffer);
console.log('Parsed to json', converted); // Parsed to json [ 5, false, 55, 'asdf' ]

Solution 2

When we are in NodeJS environment we have much better options than, Buffer.from(JSON.stringify(data)).

Performance wise JSON.stringify + Buffer.from() is ok, but will not work out if the object contains ArrayBuffer, and if done, then very inefficient.

Best way for pure NodeJS based environment

Use "v8 Serialization API" Node JS v8 docs

Its easy to use and built into the Node.js binary.

Its fastest and most space efficient among all the serializers of its kind.

const { serialize, deserialize } = require("v8")

const photo = {
   name: "rabbit",
   height: 1220,
   width: 1440,
   tinyThumbnail: jpegFileAsBuffer,
   mediumThumbnail: anotherJpegFileAsBuffer,
   description: "some string data",
   metaData: {
     tags: ["rabbit", "animal", "runner"],
     type: "image/jpeg"
   }
}

const photoSerializedAsBuffer = serialize(photo)

const deserialisedBack = deserialize(photo)

But only issue is, this only works for NodeJS. And C++ if you wish to use "v8" library(I personally not a fan of doing that).

For multi platform support

Use "bson" (MongoDB BSON)

Performance wise its close to v8 parser, but it can be adapted in all platforms where MongoDB is supported, NodeJS, JS in web, Java, C++, rust, ruby, python....

The usage is exactly like v8 serialization API

const { serialize, deserialize } = require("bson")

const photo = {
   name: "rabbit",
   height: 1220,
   width: 1440,
   tinyThumbnail: jpegFileAsBuffer,
   mediumThumbnail: anotherJpegFileAsBuffer,
   description: "some string data",
   metaData: {
     tags: ["rabbit", "animal", "runner"],
     type: "image/jpeg"
   }
}

const photoSerializedAsBuffer = serialize(photo)

const deserialisedBack = deserialize(photo)

But it can get difficult when BSON types kick in. But this should not be an issue if the object structure is know, and its unlikely that one may not know the object structure while dealing with cross platform stuff.

However a quick, solution for that in NodeJS is to use bson-buffer

At last

This not a full proof solution, but works great for NodeJS and planning to soon launch this for web JS.

tabular-json-node

And due to its simple tabular structure we can support this is other platforms too. Feel free to connect if anyone want to collaborate on this.

Comments

[dandavis]

you would have to convert the Buffer to a Blob to send() it on a Socket anyway, and gzip should take care of JSON repetition, which arrays have little of anyway.

[Lawrence Douglas]

@dandavis: Why would it have to be a blob? Can't you do connection.binaryType = "arraybuffer";?

[dandavis]

hmm maybe, i didn't know that was a Socket option, last time i got dirty, only Blobs were working in my project...

[peteb]

Have you tried Buffer.from([5, false, 55, 'asdf'], 'binary');?

[Lawrence Douglas]

@peteb: I don't think that works with the strings.

[Danibix]

new Buffer() is currently deprecated (even if it's still working in node 8.2). The supported method is Buffer.from(<string>)

[B''H Bi'ezras -- Boruch Hashem]

Hi what if my JSON object has 1)a file name (string) and 2)theactual binary data, if I were to JSON.stringify the whole thing, the ArrayBuffer (for me I'm using browser javasript) would be lost in the stringify. Is there a way to convert the JSON to an arraybuffer directly, or add a header to the arraybuffer?

[Rachael O'James]

Best solution of this task I've ever seen!