Node.js Buffers

Node.js - Buffers

Pure JavaScript is optimized for Unicode strings but traditionally lacked robust mechanisms for handling raw binary data. Interacting with network protocols (TCP, UDP), file systems, or image data often requires working directly with sequences of bytes (octets). Node.js addresses this need with the built-in Buffer class.

Buffer instances represent fixed-size chunks of memory allocated outside the V8 JavaScript engine heap. They are similar to arrays of integers, where each integer represents a byte (0-255), but with specific methods for manipulating binary data.

The Buffer class is a global type, meaning you don’t need to explicitly require('buffer') to use it.

Creating Buffers (Modern & Safe Methods)

The older new Buffer() constructor is deprecated and unsafe because it could allocate uninitialized memory, potentially exposing sensitive data. Always use the modern static methods:

• Buffer.alloc(size[, fill[, encoding]]): Creates a new, initialized (zero-filled) Buffer of the specified size in bytes. This is the safest and generally recommended method for creating a Buffer when you only know the size beforehand.
• Buffer.allocUnsafe(size): Creates a new, uninitialized Buffer of the specified size. This is faster than Buffer.alloc() but the memory segment might contain old data. Only use this if you are certain you will completely overwrite the Buffer immediately or if performance is absolutely critical.
• Buffer.from(data[, encoding]): Creates a new Buffer containing a copy of the provided data. data can be a string, another Buffer, an ArrayBuffer, a TypedArray, or an array of byte values (0-255).

Method 1: Buffer.alloc()

Creates a zero-filled Buffer of 10 bytes:

const buf = Buffer.alloc(10);
console.log(buf); // <Buffer 00 00 00 00 00 00 00 00 00 00>

Method 2: Buffer.from() with an Array

Creates a Buffer from an array of byte values:

const buf = Buffer.from([10, 20, 30, 40, 50]);
console.log(buf);       // <Buffer 0a 14 1e 28 32>
console.log(buf[0]);    // 10
console.log(buf[4]);    // 50

Method 3: Buffer.from() with a String

Creates a Buffer from a string, using a specified encoding:

// Using default UTF-8 encoding
const bufUtf8 = Buffer.from('Hello World');
console.log(bufUtf8); // <Buffer 48 65 6c 6c 6f 20 57 6f 72 6c 64>

// Using 'hex' encoding
const bufHex = Buffer.from('48656c6c6f', 'hex');
console.log(bufHex.toString('utf8')); // Hello

// Using 'base64' encoding
const bufBase64 = Buffer.from('SGVsbG8gV29ybGQ=', 'base64');
console.log(bufBase64.toString('utf8')); // Hello World

Common encodings include 'utf8', 'utf16le', 'latin1', 'ascii', 'hex', 'base64', 'base64url', 'ucs2'.

Writing to Buffers

Syntax

Use the buf.write() method to write string data into an existing buffer:

buf.write(string[, offset][, length][, encoding])

Parameters

• string: The string data to write into the buffer.
• offset: The index within the buffer to start writing at. Defaults to 0.
• length: The maximum number of bytes to write. Defaults to buffer.length - offset.
• encoding: The encoding of the input string. Defaults to ‘utf8’.

Return Value

Returns the number of bytes written. If the buffer doesn’t have enough space for the entire string (considering offset and length), only a partial string will be written.

Example

const buf = Buffer.alloc(20);
const len = buf.write('Node.js Learning');

console.log(`Bytes written: ${len}`); // Output: Bytes written: 16
console.log(buf.toString());         // Output: Node.js Learning

// Write starting at offset 8
const len2 = buf.write('Buffers', 8, 7, 'utf8'); // Write 'Buffers' over 'Learning'
console.log(`Bytes written (part 2): ${len2}`); // Output: Bytes written (part 2): 7
console.log(buf.toString());                 // Output: Node.js Buffersng

Reading from Buffers

Syntax

Use buf.toString() to decode buffer data into a string:

buf.toString([encoding][, start][, end])

Parameters

• encoding: The encoding to use for decoding. Defaults to ‘utf8’.
• start: The starting index within the buffer to begin decoding. Defaults to 0.
• end: The ending index (exclusive) within the buffer to stop decoding. Defaults to buffer.length.

Return Value

Returns the decoded string representation of the buffer’s data (or a portion of it).

Example

const buf = Buffer.alloc(26);
for (let i = 0; i < 26; i++) {
  buf[i] = i + 97; // Fill with ASCII 'a' through 'z'
}

console.log(buf.toString('ascii'));      // Outputs: abcdefghijklmnopqrstuvwxyz
console.log(buf.toString('ascii', 0, 5)); // Outputs: abcde
console.log(buf.toString('utf8', 5, 10)); // Outputs: fghij (UTF8 and ASCII are compatible here)
console.log(buf.toString(undefined, 10, 15)); // Encoding defaults to 'utf8', outputs: klmno

Convert Buffer to JSON

Syntax

The buf.toJSON() method provides a JSON representation suitable for serialization.

buf.toJSON()

Return Value

Returns an object with { type: 'Buffer', data: [array_of_byte_values] }. Note that JSON.stringify() implicitly calls this method when stringifying a Buffer instance.

Example

const buf = Buffer.from('Hello');
const json = buf.toJSON();

console.log(json);
// Output: { type: 'Buffer', data: [ 72, 101, 108, 108, 111 ] }

const jsonString = JSON.stringify(buf);
console.log(jsonString);
// Output: "{\"type\":\"Buffer\",\"data\":[72,101,108,108,111]}"

Concatenate Buffers

Syntax

Use the static Buffer.concat() method to combine multiple Buffers into a single new Buffer.

Buffer.concat(list[, totalLength])

Parameters

• list: An array of Buffer instances to concatenate.
• totalLength (Optional): The total length of the combined buffers. Providing this can improve performance as it avoids an extra loop to calculate the length.

Return Value

Returns a new Buffer containing the concatenated data from the buffers in the list.

Example

const buffer1 = Buffer.from('Node.js ');
const buffer2 = Buffer.from('is awesome!');
const buffer3 = Buffer.concat([buffer1, buffer2]);

console.log("buffer3 content: " + buffer3.toString());
// Output: buffer3 content: Node.js is awesome!

Compare Buffers

Syntax

Use buf.compare(otherBuffer) to compare the contents of two buffers lexicographically.

buf.compare(otherBuffer);

Parameters

• otherBuffer: The other Buffer instance to compare against buf.

Return Value

Returns: 0 if buf is equal to otherBuffer, 1 if buf should come after otherBuffer when sorted, -1 if buf should come before otherBuffer.

Example

const buffer1 = Buffer.from('ABC');
const buffer2 = Buffer.from('ABCD');
const buffer3 = Buffer.from('ABC');

console.log(buffer1.compare(buffer2)); // Output: -1 (buffer1 comes before buffer2)
console.log(buffer2.compare(buffer1)); // Output: 1  (buffer2 comes after buffer1)
console.log(buffer1.compare(buffer3)); // Output: 0  (buffer1 is the same as buffer3)

// Useful for sorting arrays of buffers
const arr = [buffer2, buffer1, buffer3];
arr.sort(Buffer.compare);
console.log(arr.map(b => b.toString())); // Output: [ 'ABC', 'ABC', 'ABCD' ]

Copy Buffer

Syntax

Use buf.copy() to copy data from one buffer (buf) into another (targetBuffer).

buf.copy(targetBuffer[, targetStart][, sourceStart][, sourceEnd])

Parameters

• targetBuffer: The destination Buffer object.
• targetStart: Index in targetBuffer to start writing to (default: 0).
• sourceStart: Index in the source buf to start copying from (default: 0).
• sourceEnd: Index in the source buf to stop copying from (exclusive, default: buf.length).

Return Value

Returns the number of bytes copied.

Example

const buffer1 = Buffer.from('Source Data');
const buffer2 = Buffer.alloc(15); // Destination buffer

// Copy entire buffer1 to buffer2
buffer1.copy(buffer2);
console.log("buffer2 content: " + buffer2.toString()); // Output: buffer2 content: Source Data

// Copy part of buffer1 ('Data') to buffer2 starting at index 3
const buffer3 = Buffer.alloc(10).fill(' '); // Fill with spaces
buffer1.copy(buffer3, 3, 7, 11); // targetStart=3, sourceStart=7, sourceEnd=11 ('Data')
console.log("buffer3 content: '" + buffer3.toString() + "'"); // Output: buffer3 content: '   Data    '

Slice Buffer

Syntax

Use buf.slice() to create a new Buffer that references a portion of the original buffer’s memory. This is a shallow copy – modifications to the slice affect the original Buffer and vice versa.

buf.slice([start][, end])

Parameters

• start: Index to begin the slice (inclusive, default: 0).
• end: Index to end the slice (exclusive, default: buffer.length).

Return Value

Returns a new Buffer instance representing the specified portion of the original buffer’s memory.

Example

const buffer1 = Buffer.from('Node.js Buffers');

// Create a slice containing 'Buffers'
const buffer2 = buffer1.slice(8, 15);
console.log("buffer2 content: " + buffer2.toString()); // Output: buffer2 content: Buffers

// Modify the slice
buffer2[0] = 0x58; // Change 'B' (0x42) to 'X' (0x58)

console.log("Modified slice: " + buffer2.toString());   // Output: Modified slice: Xuffers
// The original buffer is also changed!
console.log("Original buffer after slice mod: " + buffer1.toString()); // Output: Original buffer after slice mod: Node.js Xuffers

Buffer Length

Syntax

Access the buf.length property to get the allocated size of the buffer in bytes.

buf.length;

Return Value

Returns the number of bytes allocated for the Buffer instance.

Example

const buffer = Buffer.from('Example');
// length is the number of bytes, not necessarily characters
console.log("buffer length: " + buffer.length); // Output: buffer length: 7

const bufAlloc = Buffer.alloc(1024);
console.log("bufAlloc length: " + bufAlloc.length); // Output: bufAlloc length: 1024

Buffer Methods Reference

The Buffer class provides many other methods for reading and writing specific numeric types (integers, floats, doubles) with different byte orders (endianness), filling buffers, checking encodings, etc. Consult the official Node.js Buffer documentation for a complete list and detailed explanations.