zstd-codec

The library provides two primary ways to interact with Zstandard:

1. Simple API: Best for small data. It requires storing the entire input and output in Emscripten's heap. The total size of (input.length + output.length) should be less than 12MiB to avoid exceeding the available heap limits.
2. Streaming API: Best for large data. It allows storing only partial input/output bytes in the Emscripten heap, making it suitable for processing data larger than the heap limits.

When upgrading from v0.0.x to v0.1.x, note the following breaking changes:

API Instantiation

Switch from direct instantiation to the callback-based ZstdCodec.run pattern.

Old (v0.0.x):

const zstd = require('zstd-codec').ZstdCodec;
const simple = new zstd.Simple();

New (v0.1.x):

const ZstdCodec = require('zstd-codec').ZstdCodec;
ZstdCodec.run(zstd => {
    const simple = new zstd.Simple();
});

Class Name Changes

Dictionary classes have been renamed:

• ZstdCompressionDict $\rightarrow$ zstd.Dict.Compression
• ZstdDecompressionDict $\rightarrow$ zstd.Dict.Decompression

Install the zstd-codec package using your preferred package manager to use Zstandard compression in Node.js or the browser.

npm install zstd-codec
# or
yarn add zstd-codec

Because the library is powered by Emscripten, you must instantiate the API objects within the ZstdCodec.run callback to ensure the module is ready.

Note: As of v0.1.x, the library uses a callback style for instantiation rather than direct constructor access.

const ZstdCodec = require('zstd-codec').ZstdCodec;
ZstdCodec.run(zstd => {
    const simple = new zstd.Simple();
    const streaming = new zstd.Streaming();
});

To run the browser-based example of zstd-codec, follow these steps to install dependencies, build the project, and serve the files using a local HTTP server.

1. Install dependencies

Use yarn to install the necessary packages:

yarn install

2. Build the project

Compile the example code:

yarn build

3. Run the local server

Navigate to the dist/ directory and start an HTTP server on port 8084 using Python.

Using Python 3.x:

cd dist/
python -m http.server 8084

Using Python 2.x:

cd dist/
python -m SimpleHTTPServer 8084

Once the server is running, access the example at http://localhost:8084.

To use the library, you must wrap your logic in the run function. This ensures the underlying C++ bindings are correctly initialized and ready before your code executes.

run(callback) takes a function that receives the zstd object containing the Generic, Simple, Streaming, and Dict namespaces.

const { run } = require('./zstd-codec.js');

run((zstd) => {
    // Your zstd logic goes here
    const simple = new zstd.Simple();
});

The Streaming API supports both single Uint8Array inputs and Iterable chunks of Uint8Arrays. You can provide a size_hint to improve performance by suggesting the expected size of the output.

// Single buffer compression
const compressed = streaming.compress(data);

// Chunked compression
// chunks: Iterable of Uint8Array
// size_hint: (optional) size hint for performance
// compression_level: (optional) default is 3
const compressed = streaming.compressChunks(chunks, size_hint, compression_level);

// Single buffer decompression
const data = streaming.decompress(compressed, size_hint);

// Chunked decompression
const data = streaming.decompressChunks(chunks, size_hint);

The Dictionary API allows you to use trained dictionaries to improve compression ratios. You must use the Simple API instance to call compressUsingDict or decompressUsingDict.

ZstdCodec.run(zstd => {
    const simple = new zstd.Simple();

    // compress using trained dictionary
    const cdict = new zstd.Dict.Compression(dict_bytes, compression_level);
    const compressed = simple.compressUsingDict(data, cdict);

    // decompress using trained dictionary
    const ddict = new zstd.Dict.Decompression(dict_bytes);
    const data = simple.decompressUsingDict(compressed, ddict);
});

The Simple API provides direct compress and decompress methods. Input data must be a Uint8Array.

// compress
// content_bytes: Uint8Array
// compression_level: (optional) default is 3
const compressed = simple.compress(data, level);

// decompress
// compressed_bytes: Uint8Array
const data = simple.decompress(compressed);

The Generic class provides utility methods for inspecting Zstd data without performing full compression/decompression.

• compressBound(content_bytes): Returns the maximum possible size of the compressed output for a given input size. Takes a Uint8Array and returns a number.
• contentSize(compressed_bytes): Returns the original uncompressed size of the provided compressed_bytes. Takes a Uint8Array and returns a number.

const { run } = require('./zstd-codec.js');

run((zstd) => {
    const generic = new zstd.Generic();
    const input = new Uint8Array([1, 2, 3]);
    
    const maxBound = generic.compressBound(input);
    // ... compress ...
    const originalSize = generic.contentSize(compressed);
});

The ZstdCodec class provides the primary interface for Zstandard compression and decompression operations. It supports three main modes of operation: simple buffer-to-buffer compression, dictionary-based compression for better efficiency with small data, and information retrieval (calculating bounds and content size).

// Example usage of ZstdCodec
ZstdCodec codec;
Vec<u8> src = { ... };
Vec<u8> dest;

// Simple compression
int result = codec.Compress(dest, src, compression_level);

// Simple decompression
int decomp_result = codec.Decompress(dest, src);

The ZstdDecompressTransform class is a Node.js Transform stream used to decompress Zstd-compressed data. It can be used within a stream pipeline to decompress chunks of data as they pass through.

Constructor Arguments

• option (object, optional): Standard Node.js Transform stream options.

const { ZstdDecompressTransform } = streams;
const decompressor = new ZstdDecompressTransform();
// Use in a pipeline: compressedStream.pipe(decompressor).pipe(readableStream);