python-zstandard

The project is currently in beta. Users should expect potential backwards incompatible changes before version 1.0 (likely in the 0.9 release), such as renaming the main module from zstd to zstandard and renaming various types and methods.

Best Practice: Pin your package version to prevent unwanted breakage when these changes occur.

Supported platforms include Linux x86_x64 and Windows x86/x86_64 (Python 3.10+). The library provides both C extension and CFFI implementations, with the CFFI bindings being mostly feature complete.

Many functions in python-zstandard accept any Python object that implements the buffer protocol. This allows the library to access the raw bytes of an object without requiring explicit conversion to bytes first.

Supported types include:

• bytes
• bytearray
• array.array
• io.BytesIO
• mmap.mmap
• memoryview

Zstandard is highly tunable. Performance characteristics depend on the settings used:

• Default (Level 3): Generally faster and provides better compression ratios than zlib on most datasets.
• Speed-tuned: Approaches lz4 speed and ratios.
• Ratio-tuned: Approaches lzma ratios and compression speed, but with much faster decompression.

Key Performance Features:

• Multi-threading: Supports multi-threaded compression for large inputs.
• Zero-copy: The library provides multiple APIs to facilitate zero-copy operations and minimize Python object creation/garbage collection overhead.
• Throughput: Capable of single-threaded throughputs exceeding 1 GB/s.

To measure performance on your specific hardware, use the bench.py script found in the source code repository.

When using one-shot (non-streaming) decompression APIs, you must provide a way to determine the output size. This is because the API requires a pre-allocated buffer to store the result.

To decompress, you must either:

1. Ensure the zstd frame header contains the decompressed size.
2. Explicitly pass the required output size to the function.
3. Specify a maximum output size.

Security Note: The library limits the maximum output size to prevent 'decompression bombs' (e.g., a small input that expands to many gigabytes), which could otherwise exhaust system memory.

Zstandard data is contained within a frame, which includes a header and an optional trailer. The header identifies the data as a zstd frame and describes the compressed content.

To optimize decompression performance, you should store the original content size within the frame. This allows the decompressor to perform a single, exact memory allocation for the output rather than repeatedly growing a buffer.

In python-zstandard, you can enable this by setting write_content_size=True when initializing a ZstdCompressor.

# Example of enabling content size writing for better decompression performance
compressor = zstd.ZstdCompressor(write_content_size=True)

A compression dictionary is used to seed the compressor state with common patterns found in your data. This is highly effective when compressing many small, similar objects (e.g., JSON documents with the same structure).

When to use: Dictionary compression is generally only beneficial for small inputs, typically data no larger than a few kilobytes. The effectiveness depends on the similarity between your data and the dictionary used.

Many APIs used by this module are marked as experimental within the upstream Zstandard project (e.g., dictionary training).

Because the Zstandard C API's evolution for experimental features is uncertain, the behavior of the underlying C API might change. However, python-zstandard mitigates this by vendoring and statically linking a specific version of the Zstandard source code. This ensures that the behavior of a specific version of python-zstandard remains constant.

Recommendation: Pin your version of python-zstandard to ensure consistent behavior and protection from upstream changes.

A context holds the configuration (like compression level) and the state for a zstd operation. In this library, ZstdCompressor and ZstdDecompressor act as wrappers around these C API contexts.

Performance Tip: Creating and destroying contexts is computationally expensive. You should reuse ZstdCompressor and ZstdDecompressor instances for multiple operations to gain performance advantages.

You can perform compression and decompression using two different modes:

1. One-shot (Simple APIs): All input is provided as a single buffer, and the entire output is returned as a single buffer. This is the simplest way to use the library for data that fits comfortably in memory.
2. Streaming: Input and output are handled in chunks via multiple function calls. This requires a stream object (a logical extension of a context) to track the state of the ongoing operation. Streaming is necessary for processing data that is too large to fit in memory at once.

The library provides two main categories of APIs depending on your data size and memory constraints:

One-Shot APIs

• Best for: Small data where the input and output sizes are known.
• Pros: Simple to use (single function call).
• Cons: Both input and output must fit in memory simultaneously. Large inputs can cause long blocking operations.

Streaming APIs

• Best for: Large datasets or data arriving in chunks.
• Pros: Does not require all data to be in memory at once; provides fine-grained control over input/output flow.
• Cons: More complex to implement than one-shot calls.
• Note: When using streaming APIs with file-like or stream objects, be aware that I/O operations (like network or disk access) can cause long pauses in the execution flow.