Convert a binary NodeJS Buffer to JavaScript ArrayBuffer
Solution 1
Instances of Buffer
are also instances of Uint8Array
in node.js 4.x and higher. Thus, the most efficient solution is to access the buf.buffer
property directly, as per https://stackoverflow.com/a/31394257/1375574. The Buffer constructor also takes an ArrayBufferView argument if you need to go the other direction.
Note that this will not create a copy, which means that writes to any ArrayBufferView will write through to the original Buffer instance.
In older versions, node.js has both ArrayBuffer as part of v8, but the Buffer class provides a more flexible API. In order to read or write to an ArrayBuffer, you only need to create a view and copy across.
From Buffer to ArrayBuffer:
function toArrayBuffer(buf) {
const ab = new ArrayBuffer(buf.length);
const view = new Uint8Array(ab);
for (let i = 0; i < buf.length; ++i) {
view[i] = buf[i];
}
return ab;
}
From ArrayBuffer to Buffer:
function toBuffer(ab) {
const buf = Buffer.alloc(ab.byteLength);
const view = new Uint8Array(ab);
for (let i = 0; i < buf.length; ++i) {
buf[i] = view[i];
}
return buf;
}
Solution 2
No dependencies, fastest, Node.js 4.x and later
Buffer
s are Uint8Array
s, so you just need to slice (copy) its region of the backing ArrayBuffer
.
// Original Buffer
let b = Buffer.alloc(512);
// Slice (copy) its segment of the underlying ArrayBuffer
let ab = b.buffer.slice(b.byteOffset, b.byteOffset + b.byteLength);
The slice
and offset stuff is required because small Buffer
s (less than 4 kB by default, half the pool size) can be views on a shared ArrayBuffer
. Without slicing, you can end up with an ArrayBuffer
containing data from another Buffer
. See explanation in the docs.
If you ultimately need a TypedArray
, you can create one without copying the data:
// Create a new view of the ArrayBuffer without copying
let ui32 = new Uint32Array(b.buffer, b.byteOffset, b.byteLength / Uint32Array.BYTES_PER_ELEMENT);
No dependencies, moderate speed, any version of Node.js
Use Martin Thomson's answer, which runs in O(n) time. (See also my replies to comments on his answer about non-optimizations. Using a DataView is slow. Even if you need to flip bytes, there are faster ways to do so.)
Dependency, fast, Node.js ≤ 0.12 or iojs 3.x
You can use https://www.npmjs.com/package/memcpy to go in either direction (Buffer to ArrayBuffer and back). It's faster than the other answers posted here and is a well-written library. Node 0.12 through iojs 3.x require ngossen's fork (see this).
Solution 3
"From ArrayBuffer to Buffer" could be done this way:
var buffer = Buffer.from( new Uint8Array(ab) );
Solution 4
A quicker way to write it
var arrayBuffer = new Uint8Array(nodeBuffer).buffer;
However, this appears to run roughly 4 times slower than the suggested toArrayBuffer function on a buffer with 1024 elements.
Solution 5
1. A Buffer
is just a view for looking into an ArrayBuffer
.
A Buffer
, in fact, is a FastBuffer
, which extends
(inherits from) Uint8Array
, which is an octet-unit view (“partial accessor”) of the actual memory, an ArrayBuffer
.
/lib/buffer.js#L65-L73
Node.js 9.4.0
class FastBuffer extends Uint8Array {
constructor(arg1, arg2, arg3) {
super(arg1, arg2, arg3);
}
}
FastBuffer.prototype.constructor = Buffer;
internalBuffer.FastBuffer = FastBuffer;
Buffer.prototype = FastBuffer.prototype;
2. The size of an ArrayBuffer
and the size of its view may vary.
Reason #1: Buffer.from(arrayBuffer[, byteOffset[, length]])
.
With Buffer.from(arrayBuffer[, byteOffset[, length]])
, you can create a Buffer
with specifying its underlying ArrayBuffer
and the view's position and size.
const test_buffer = Buffer.from(new ArrayBuffer(50), 40, 10);
console.info(test_buffer.buffer.byteLength); // 50; the size of the memory.
console.info(test_buffer.length); // 10; the size of the view.
Reason #2: FastBuffer
's memory allocation.
It allocates the memory in two different ways depending on the size.
- If the size is less than the half of the size of a memory pool and is not 0 (“small”): it makes use of a memory pool to prepare the required memory.
-
Else: it creates a dedicated
ArrayBuffer
that exactly fits the required memory.
/lib/buffer.js#L306-L320
Node.js 9.4.0
function allocate(size) {
if (size <= 0) {
return new FastBuffer();
}
if (size < (Buffer.poolSize >>> 1)) {
if (size > (poolSize - poolOffset))
createPool();
var b = new FastBuffer(allocPool, poolOffset, size);
poolOffset += size;
alignPool();
return b;
} else {
return createUnsafeBuffer(size);
}
}
📜/lib/buffer.js#L98-L100
Node.js 9.4.0
function createUnsafeBuffer(size) {
return new FastBuffer(createUnsafeArrayBuffer(size));
}
What do you mean by a “memory pool?”
A memory pool is a fixed-size pre-allocated memory block for keeping small-size memory chunks for Buffer
s. Using it keeps the small-size memory chunks tightly together, so prevents fragmentation caused by separate management (allocation and deallocation) of small-size memory chunks.
In this case, the memory pools are ArrayBuffer
s whose size is 8 KiB by default, which is specified in Buffer.poolSize
. When it is to provide a small-size memory chunk for a Buffer
, it checks if the last memory pool has enough available memory to handle this; if so, it creates a Buffer
that “views” the given partial chunk of the memory pool, otherwise, it creates a new memory pool and so on.
You can access the underlying ArrayBuffer
of a Buffer
. The Buffer
's buffer
property (that is, inherited from Uint8Array
) holds it. A “small” Buffer
's buffer
property is an ArrayBuffer
that represents the entire memory pool. So in this case, the ArrayBuffer
and the Buffer
varies in size.
const zero_sized_buffer = Buffer.allocUnsafe(0);
const small_buffer = Buffer.from([0xC0, 0xFF, 0xEE]);
const big_buffer = Buffer.allocUnsafe(Buffer.poolSize >>> 1);
// A `Buffer`'s `length` property holds the size, in octets, of the view.
// An `ArrayBuffer`'s `byteLength` property holds the size, in octets, of its data.
console.info(zero_sized_buffer.length); /// 0; the view's size.
console.info(zero_sized_buffer.buffer.byteLength); /// 0; the memory..'s size.
console.info(Buffer.poolSize); /// 8192; a memory pool's size.
console.info(small_buffer.length); /// 3; the view's size.
console.info(small_buffer.buffer.byteLength); /// 8192; the memory pool's size.
console.info(Buffer.poolSize); /// 8192; a memory pool's size.
console.info(big_buffer.length); /// 4096; the view's size.
console.info(big_buffer.buffer.byteLength); /// 4096; the memory's size.
console.info(Buffer.poolSize); /// 8192; a memory pool's size.
3. So we need to extract the memory it “views.”
An ArrayBuffer
is fixed in size, so we need to extract it out by making a copy of the part. To do this, we use Buffer
's byteOffset
property and length
property, which are inherited from Uint8Array
, and the ArrayBuffer.prototype.slice
method, which makes a copy of a part of an ArrayBuffer
. The slice()
-ing method herein was inspired by @ZachB.
const test_buffer = Buffer.from(new ArrayBuffer(10));
const zero_sized_buffer = Buffer.allocUnsafe(0);
const small_buffer = Buffer.from([0xC0, 0xFF, 0xEE]);
const big_buffer = Buffer.allocUnsafe(Buffer.poolSize >>> 1);
function extract_arraybuffer(buf)
{
// You may use the `byteLength` property instead of the `length` one.
return buf.buffer.slice(buf.byteOffset, buf.byteOffset + buf.length);
}
// A copy -
const test_arraybuffer = extract_arraybuffer(test_buffer); // of the memory.
const zero_sized_arraybuffer = extract_arraybuffer(zero_sized_buffer); // of the... void.
const small_arraybuffer = extract_arraybuffer(small_buffer); // of the part of the memory.
const big_arraybuffer = extract_arraybuffer(big_buffer); // of the memory.
console.info(test_arraybuffer.byteLength); // 10
console.info(zero_sized_arraybuffer.byteLength); // 0
console.info(small_arraybuffer.byteLength); // 3
console.info(big_arraybuffer.byteLength); // 4096
4. Performance improvement
If you're to use the results as read-only, or it is okay to modify the input Buffer
s' contents, you can avoid unnecessary memory copying.
const test_buffer = Buffer.from(new ArrayBuffer(10));
const zero_sized_buffer = Buffer.allocUnsafe(0);
const small_buffer = Buffer.from([0xC0, 0xFF, 0xEE]);
const big_buffer = Buffer.allocUnsafe(Buffer.poolSize >>> 1);
function obtain_arraybuffer(buf)
{
if(buf.length === buf.buffer.byteLength)
{
return buf.buffer;
} // else:
// You may use the `byteLength` property instead of the `length` one.
return buf.subarray(0, buf.length);
}
// Its underlying `ArrayBuffer`.
const test_arraybuffer = obtain_arraybuffer(test_buffer);
// Just a zero-sized `ArrayBuffer`.
const zero_sized_arraybuffer = obtain_arraybuffer(zero_sized_buffer);
// A copy of the part of the memory.
const small_arraybuffer = obtain_arraybuffer(small_buffer);
// Its underlying `ArrayBuffer`.
const big_arraybuffer = obtain_arraybuffer(big_buffer);
console.info(test_arraybuffer.byteLength); // 10
console.info(zero_sized_arraybuffer.byteLength); // 0
console.info(small_arraybuffer.byteLength); // 3
console.info(big_arraybuffer.byteLength); // 4096
Related videos on Youtube
Drake Amara
Updated on February 16, 2022Comments
-
Drake Amara over 2 years
How can I convert a NodeJS binary buffer into a JavaScript ArrayBuffer?
-
Chris Biscardi over 12 yearsI'm curious as to why you would need to do this?
-
fbstj over 12 yearsa good example would be writing a library that worked with File's in browsers and also for NodeJS files?
-
OrangeDog almost 11 yearsor using a browser library in NodeJS
-
nponeccop over 10 yearsAnother reason is that a float takes too many bytes of RAM when stored in an
Array
. So to store many floats you needFloat32Array
where it takes 4 bytes. And if you want quick serialization of those floats to a file you need aBuffer
, as serializing to JSON takes ages. -
Felix Crazzolara almost 6 yearsI want to know exactly the same thing to send generic data using WebRTC and it's unbelievable that so many answers here have so many likes, but don't answer the actual question...
-
William Entriken almost 4 years
const file = fs.readFileSync(filePath);
, so how do I use this?... 30 minutes later, wow I miss C.
-
-
Triang3l over 11 yearsI'd also recommend you to optimize this by copying integers when possible using DataView. Until
size&0xfffffffe
, copy 32-bit integers, then, if there's 1 byte remaining, copy 8-bit integer, if 2 bytes, copy 16-bit integer, and if 3 bytes, copy 16-bit and 8-bit integer. -
OrangeDog almost 11 yearsSee kraag22's answer for a simpler implementation of half of this.
-
pospi almost 10 yearsHave tested Buffer -> ArrayBuffer with a module intended for browser use and it is working brilliantly. Thanks!
-
aleclarson almost 10 years
-
ZachB over 9 years@SiPlus are you sure that's an optimization? The DataView methods (e.g.
getInt32
) are exceptionally slow compared to accessing array views because of a ton of type-checking that goes on. Here, reading from a DataView is 98% slower, which you won't make up for through 4x fewer iterations you'd get from your approach: jsperf.com/haakons-test -
Triang3l over 9 years@ZachB Yes, aligning the offset and using Uint32Array should be much better.
-
ZachB over 9 years@SiPlus grin "should be" and "is" better are not the same; always test "optimizations" before assuming they're optimizations. First off, you actually can't create a DataView from a node Buffer. However, you can use
buffer.readUInt32LE
. This is still 80% slower than Martin's method, see here for the benchmark. As for aligning offsets, I don't know what you mean. Typed array views must always be memory-aligned or the engine will throw an error. The user has no control here. (Node seems to allow non-aligned access to its Buffers.) -
Triang3l over 9 years@ZachB I was talking about the typed arrays in browser. Copy the first 0-3 bytes, copy the ints, copy the last 0-3 bytes.
-
ZachB over 9 years@SiPlus Huh? The question is about node Buffers, which do not exist in browsers. My statements remain true that byte-wise operations are faster than using a read method, anywhere.
-
ChrisV almost 9 yearsLate addition: @trevnorris says "starting in [V8] 4.3 Buffers are backed by Uint8Array", so possibly this is faster now...
-
ZachB almost 9 yearsThat's why Martin Thomson's answer uses Uint8Array -- it is agnostic to the size of the elements. The
Buffer.read*
methods are all slow, also. -
ZachB almost 9 yearsMultiple typed array views can reference the same ArrayBuffer using the same memory. Each value in a Buffer is one byte, so you need to put it into an array with element size of 1 byte. You can use Martin's method, then make a new Float64Array using the same arraybuffer in the constructor.
-
jcalfee314 over 8 yearsThe 1st example, your returning ab and not view?
-
Pawel Veselov over 8 yearsIt doesn't compile again node > 0.12
-
ZachB over 8 yearsUse ngossen's fork: github.com/dcodeIO/node-memcpy/pull/6. See also my new answer if you're using node 4+.
-
ZachB over 8 yearsThis looks like a specialized (and slow) serialization-based method, not a generic method for converting to/from Buffer/ArrayBuffer?
-
ZachB over 8 yearsSee my answer for the safe way to do this.
-
Miguel Valentine over 8 years@ZachB it is generic method for V5.0.0+[only] = =.
-
ZachB over 8 years
toArrayBuffer(new Buffer([1,2,3]))
->['01', '02', '03']
-- this is returning an array of strings, not integers/bytes. -
Miguel Valentine over 8 years@ZachB return array ->return list. i fix int->string for stdout
-
ZachB over 8 yearsIn that case it's the same as stackoverflow.com/a/19544002/1218408, and still without necessary the byte offset checks in stackoverflow.com/a/31394257/1218408.
-
svenyonson over 8 yearstoArrayBuffer should return the ArrayBuffer instance (ab), not the view. The caller needs to be able to create whatever view they need from ArrayBuffer. You cannot to this from Uint8Array
-
daksh_019 over 8 yearsTested it with v5.6.0 and it was the fastest
-
Alexander Gonchiy about 8 yearsThat's the opposite of what OP wanted.
-
Andi Giga almost 8 yearsWhy is
ab
returned? There is nothing done withab
? I always get{}
as a result. -
koceeng over 7 yearsWhile this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes
-
Dlabz about 7 yearsMy wording might not sound very official, but it does provide enough information to recreate the solution. The solution relies on JavaScript Proxy Object to wrap a native NodeJS Buffer with getters and setters used by TypedArrays. This makes the Buffer instance compatible with any library that requires Typed Array interface. This is the answer original poster was hoping for, but feel free to dismiss it as it doesn't fit your academic/corporate lingo. See if I care.
-
Maciej Krawczyk about 7 yearsBut that's what I wanted googling my problem and glad I've found the solution.
-
Benny Neugebauer about 7 yearsThis only works because instances of
Buffer
are also instances ofUint8Array
in Node.js 4.x and higher. For lower Node.js versions you have to implement atoArrayBuffer
function. -
Константин Ван over 6 years‘The
slice()
method returns a newArrayBuffer
whose contents are a copy of thisArrayBuffer
's bytes from begin, inclusive, up to end, exclusive.’ - MDNArrayBuffer.prototype.slice()
-
Константин Ван over 6 yearsDOWNVOTED,
slice()
-ing anArrayBuffer
DOES make a copy. -
Константин Ван over 6 yearsWhere were the
.byteLength
and.byteOffset
documented? -
ZachB over 6 years@K._ Those properties are inherited from TypedArray: developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/… and developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/…
-
tjmehta over 6 yearsTypedArrays have a
buffer
property which is anArrayBuffer
.. so there is no need to iterate through each index. -
Tustin2121 about 6 yearsThis is all well and good... but did you actually answer OP's question? If you did, it's buried...
-
Marcin Czenko almost 6 yearsGreat answer! In
obtain_arraybuffer
:buf.buffer.subarray
does not seem to exist. Did you meanbuf.buffer.slice
here? -
Константин Ван almost 6 years@everydayproductive Thank you. As you can see in the edit history, I actually used
ArrayBuffer.prototype.slice
and later modified it toUint8Array.prototype.subarray
. Oh, and I did it wrong. Probably got a bit confused back then. It's all good right now thanks to you. -
Alexey Sh. over 5 years
var ab = b.buffer.slice(b.byteOffset, b.byteOffset + b.byteLength);
saved my day -
Mike Frysinger about 4 yearsUsing
slice()
is not O(1), it's O(n) because it creates a copy: developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/… -
ZachB about 4 years@MikeFrysinger oops, thanks. Fixed and refreshed answer.
-
Gilbert over 2 yearsWhy do you call a UintArray a view?
-
Martin Thomson over 2 yearsUint8Array is one of several different views of a portion of memory.