DOCSEARCHbyAlgolia
Submit Docs

ZXC Documentation

repository·main·Indexed 19 days ago

https://github.com/hellobertrand/zxc

ZXC is an asymmetric lossless compression C library optimized for ultra-fast decompression and "Write Once, Read Many" (WORM) workloads. It features high decode throughput, $O(1)$ random-access decompression via a built-in seek table, and hand-tuned SIMD support for x86_64, ARM, RISC-V, POWER, s390x, and i386. The library provides seven compression levels ranging from Max Speed (-1) to Ultra (-7) and offers official bindings for Rust, Python, Node.js, and Go.

Tokens
69.5K
Snippets
207
Records
285
Agent score
64%

What's inside ZXC

  1. Overview of ZXC compression

    main

    ZXC is a lossless compression C library designed for asymmetric efficiency. It is optimized for "Write Once, Read Many" (WORM) workloads, such as content delivery, game assets, app bundles, and firmware updates.

    Key characteristics include:

    • High Decode Throughput: Designed to be faster than LZ4 at a smaller size, especially on ARM64 architectures.
    • Asymmetric Design: Offloads complexity to the encoder (build time) to maximize decompression speed (run time).
    • Cross-platform Support: Runs on x86_64, ARM64, ARMv7, ARMv6, RISC-V, POWER, s390x, and i386 with hand-tuned SIMD (SSE2/AVX2/AVX-512 on x86, NEON on ARMv8+).
    • Seekable: Includes a built-in seek table for $O(1)$ random-access decompression.
    • Production-ready: Thread-safe API, verified via extensive fuzzing, and available via multiple package managers.

  2. Key features of ZXC Python Bindings

    main

    The ZXC Python bindings are optimized for Write Once, Read Many workloads (such as ML datasets, game assets, and caches) and provide the following capabilities:

    • Fast Decompression: Optimized specifically for read-heavy workloads.
    • Buffer Protocol Support: Compatible with bytes, bytearray, memoryview, and NumPy arrays.
    • GIL Release: The compression and decompression processes release the Global Interpreter Lock (GIL), allowing for true parallelism when using Python threads.
    • Stream Helpers: Includes utilities to compress and decompress file-like objects.

  3. Configure Buffer Descriptors for Push Streaming

    main

    The Push Streaming API uses two descriptor structures to manage memory windows. The library reads from/writes to the memory pointed to by src or dst and advances the pos field by the number of bytes consumed or produced.

    Important Memory Safety Note: The entire window [dst+pos .. dst+size) is considered writable scratch space. The decoder may perform speculative (wild-copy) stores into this window. Consequently, bytes beyond the final pos are unspecified; never rely on or keep live data inside the declared capacity of the buffer.

    Buffer Types:

    • zxc_inbuf_t: Used for input data. Contains src (pointer), size (total capacity), and pos (current offset).
    • zxc_outbuf_t: Used for output data. Contains dst (pointer), size (total capacity), and pos (current offset).
    typedef struct {
    const void* src;
    size_t      size;
    size_t      pos;   // advanced by the library
} zxc_inbuf_t;

typedef struct {
    void*  dst;
    size_t size;
    size_t pos;        // advanced by the library
} zxc_outbuf_t;

  4. Understand the ZXC asymmetric compression model

    main

    ZXC is an asymmetric lossless codec designed for 'Write-Once, Read-Many' (WORM) workflows, such as mobile gaming assets or firmware updates.

    Unlike symmetric codecs (like LZ4) that aim for similar speeds in both compression and decompression, ZXC uses a computationally intensive encoder to produce a bitstream optimized for maximum decompression throughput. This offloads complexity from the consumer (e.g., an ARM-based mobile device) to the producer (e.g., an x86 build server), resulting in significantly faster decompression and reduced battery consumption on end-user devices.

  5. Understand the ZXC Asynchronous Decompression Pipeline

    main

    ZXC's decompression is optimized for speed through parallel decoding and SIMD acceleration. The pipeline consists of:

    1. Header Parsing (Main Thread): The main thread scans block headers to determine chunk boundaries and payload sizes.
    2. Dispatch: Compressed payloads are distributed to a worker job queue.
    3. Parallel Decoding (Worker Threads): Workers decode chunks into pre-allocated output buffers.
      • Fast Path: When the output buffer has sufficient margin, the decoder utilizes "wild copies" (16-byte SIMD stores) to bypass bounds checking for maximum throughput.
    4. Serialization: Decompressed blocks are committed to the output stream sequentially to maintain file integrity.

  6. How Pivoted Coding Huffman (PivCo) works

    main

    In ZXC format v7, entropy decoding is optimized via Pivoted Coding Huffman (PivCo). Unlike classical Huffman decoding which is a serial bit-chain dependency, PivCo uses a level-ordered layout that allows for data-parallel decoding using SIMD instructions.

    Key characteristics:

    • Transposed Layout: Instead of symbol-after-symbol, the wire format stores branch bits for every internal node of the code tree in BFS order.
    • SIMD Acceleration: Decoding is implemented as a series of data-parallel list merges using byte shuffles (e.g., TBL on NEON, pshufb on SSSE3/AVX2).
    • Performance: On Apple M3, this achieves ~3.1 GB/s single-threaded, a ~77% improvement over traditional 4-stream classic decoders.

  7. Use Shared-table Huffman literals (enc_lit=3)

    main

    The enc_lit=3 mode is used for dictionary-compressed archives (where HAS_DICTIONARY is set in the file header).

    Key differences from standard Huffman (enc_lit=2):

    • The 128-byte inline code lengths header is omitted.
    • The code lengths are instead retrieved from the shared literal table carried by the .zxd dictionary.

    Requirements:

    • The archive's dict_id must match the attached dictionary.
    • Decoders MUST reject enc_lit=3 sections with ZXC_ERROR_DICT_REQUIRED if no dictionary table is attached.

  8. Understand the ZXD Dictionary File Format

    main

    A .zxd file is a dictionary file used by ZXC to provide a pre-filled LZ77 window and a shared Huffman table for compression. The format consists of a 16-byte header, the raw dictionary content, and a 128-byte shared Huffman table.

    Important: The dict_id stored in the .zxd header must match the dict_id stored in the file header of any .zxc archive that uses this dictionary. This binds the specific (content, table) pair to the archive.

    File Layout

    OffsetSizeDescription
    0x0016 bytesDictionary Header
    0x10VariableDictionary Content (Raw bytes)
    0x10 + content_size128 bytesShared Huffman Table

    Dictionary Header Fields (16 bytes)

    FieldSizeDescription
    Magic Word4 bytesLittle-Endian 0x9CB0D1C7 (identifies .zxd)
    Version1 byteDictionary format version
    Flags1 byteBits 0..3: checksum algorithm ID (e.g., 0 for RapidHash); Bits 4..7: reserved
    Content Size2 bytesLittle-Endian size of the raw content
    dict_id4 bytesLittle-Endian ID used to bind the dictionary to a .zxc archive
    Reserved2 bytesReserved
    Header CRC162 bytesLittle-Endian CRC16 computed over the 16-byte header (with bytes 0x0C..0x0F zeroed)
    /* Example structure of a 149-byte .zxd file containing 'hello' */
// 0x00..0x0F: Dictionary Header (16 bytes)
// 0x10..0x14: Dictionary Content (5 bytes)
// 0x15..0x94: Shared Huffman Table (128 bytes)

  9. Understand the ZXC File Format Structure

    main

    The ZXC file format is a block-based, robust format designed for parallel processing and random access. A complete ZXC stream consists of three main parts:

    1. File Header (16 bytes): Identifies the format, version, chunk size, and global configuration (flags, dictionary ID).
    2. Data Blocks: A sequence of blocks, each starting with an 8-byte Block Header followed by a payload (and an optional 4-byte checksum).
      • Type 0 (RAW): Uncompressed data.
      • Type 1 (GLO): Generic Low-velocity blocks for general compression/speed balance.
      • Type 2 (GHI): Generic High-velocity blocks optimized for maximum decompression throughput.
      • Type 255 (EOF): End-of-file marker.
    3. File Footer (12 bytes): Contains the total uncompressed source size and a global stream checksum.

  10. How to train and use ZXC dictionaries

    main

    ZXC provides a Dictionary API (<zxc_dict.h>) for improving compression ratios using representative samples.

    The Simple Workflow:

    1. Train: Use zxc_dict_train() to create a .zxd dictionary file from a corpus of samples. This function handles content training, Huffman table training, and serialization in one call.
    2. Load: Use zxc_dict_load() to parse a .zxd file. This is a zero-copy operation where the returned pointers point directly into the input buffer.
    3. Use: Pass the resulting content and Huffman table pointers into your compression/decompression options.

    For seekable archives, you must attach the dictionary to the handle using zxc_seekable_set_dict() before performing any range decompression.

    // Simple dictionary creation
void* zxd_buf;
int64_t bytes_written = zxc_dict_train(samples, sample_sizes, n_samples, zxd_buf, zxd_capacity);

// Attaching to a seekable handle
zxc_seekable_set_dict(seekable_handle, dict_content, dict_size, dict_huf_table);

  11. ZXC File Format Overview and Layout

    main

    A ZXC compressed file (version 7) is a sequence of blocks terminated by an EOF block and a footer. The format uses little-endian byte order for all multi-byte integers.

    Full File Layout:

    1. File Header (16 bytes)
    2. Blocks (Sequence of blocks, each with a header, payload, and optional 4B CRC32)
    3. EOF Block (8 bytes, type=255, comp_size=0)
    4. SEK Block (Optional, provides a table of contents for random access)
    5. File Footer (12 bytes)
    +----------------------+ 16 bytes
| File Header          |
+----------------------+
| Block #0             |
|  - 8B Block Header   |
|  - Block Payload     |
|  - Optional 4B CRC32 |
+----------------------+
| Block #1             |
|  ...                 |
+----------------------+
| EOF Block            | 8 bytes (type=255, comp_size=0)
+----------------------+
| SEK Block (Optional) | table of contents for random access
+----------------------+
| File Footer          | 12 bytes
+----------------------+

  12. Use dictionaries with zxc for better compression

    main

    For workloads using small blocks (4K–128K), pre-trained dictionaries (.zxd files) significantly improve compression ratios by prefilling the LZ77 window.

    Training a dictionary

    Use the --train mode with training samples. The output is a .zxd file. If -o is used with a directory, the file is saved as dictionary_<dict_id>.zxd.

    Using a dictionary

    • Compression: Pass the dictionary file with -D <FILE>.
    • Decompression: The dictionary is mandatory. You must provide the exact dictionary used during compression via -D <FILE>, or decompression will fail. There is no automatic lookup.

    Note: The .zxd extension is cosmetic; files are identified by a magic word.

    # Train a dictionary from samples
zxc --train -o dicts/ samples/*.json

# Compress using a specific dictionary
zxc -B 4K -D dicts/dictionary_bc46eec1.zxd input.json

# Decompress using the required dictionary
zxc -d -D dicts/dictionary_bc46eec1.zxd input.json.zxc