klauspost/compress

The zstd package provides a pure Go implementation of the Zstandard compression algorithm. It is designed for high-performance compression and decompression, offering a wide range of compression/speed trade-offs.

Key considerations:

• Architecture: The package is heavily optimized for 64-bit processors; performance may be significantly lower on 32-bit processors.
• Pure Go: It is a pure Go implementation. You can use noasm and nounsafe build tags to disable specific features if required.
• Seekable Streams: This package does not natively support seekable Zstandard streams. For that functionality, use github.com/SaveTheRbtz/zstd-seekable-format-go.

The klauspost/compress repository provides a collection of high-performance compression algorithms and optimized implementations for Go. Supported algorithms and packages include:

• zstandard: Pure Go implementation of Zstandard compression and decompression.
• S2: A high-performance replacement for Snappy.
• deflate: Optimized packages that serve as drop-in replacements for gzip, zip, and zlib.
• snappy: A drop-in replacement for github.com/golang/snappy with improved compression and concurrent stream support.
• huff0 and FSE: Implementations for raw entropy encoding.
• gzhttp: Client and server wrappers for efficient handling of gzipped/zstd HTTP requests.
• pgzip: A separate package providing a high-speed parallel gzip implementation.

S2 provides different compression modes that impact decompression speed:

• Standard S2: Optimized for fast decompression. It prefers longer matches (typically 6 bytes or longer), which improves decompression speed at the cost of slightly lower compression ratios.
• "better" mode: Actively looks for shorter matches to achieve a smaller representation. This mode results in decompression speeds similar to Snappy.

S2 is designed as a high-speed replacement for Snappy. It typically provides 5-20% better compression and 25-40% higher throughput (single-threaded) compared to the Snappy Go implementation.

Key performance characteristics:

• Concurrency: Streams are concurrently compressed and distributed across all available CPU cores for maximum throughput.
• "Better" Compression Mode: A mode is available to trade some speed for improved compression gains. Data compressed in this mode remains fully compatible with the standard S2 decoder.
• Data Types: Machine-generated data often sees the largest compression boosts (up to 35% smaller than Snappy), while incompressible/random data sees the smallest speedups due to synchronization overhead.

The fse package provides Finite State Entropy (tANS) encoding and decoding for byte blocks. It is designed for fast, near-optimal symbol encoding, similar to the implementation used in Zstandard.

Key Characteristics:

• Block-based: It compresses single independent blocks. It does not perform multi-byte dictionary coding (like LZ coders) but can be used as a secondary step after compressors like Snappy.
• No Integrity Checks: The package does not provide built-in integrity checks. The caller is responsible for tracking block sizes and implementing checksums to ensure data validity.
• Error Handling: Some errors are expected during normal operation. You must handle ErrIncompressible (input is too hard to compress) and ErrUseRLE (input consists of a single repeated byte value).

When compressing individual blocks, S2 does not perform concurrent compression (similar to Snappy).

Key characteristics of block compression:

• Incompressible blocks: Unlike Snappy, where incompressible blocks can grow significantly, S2 ensures incompressible blocks will be at most 10 bytes larger than the input.
• Performance: S2 generally delivers higher throughput and better compression than Snappy. When configured to output Snappy-compatible data, it still maintains higher throughput and better compression ratios.

To improve compression/decompression speeds and save space, you can reuse tables from previous blocks. This is managed via the ReusePolicy on a Scratch object.

Implementation Details:

• Policy Control: You can set and alter the ReusePolicy between blocks using the Scratch object.
• Manual Tracking: The package does not store reuse information in the output block. The caller is responsible for recording whether ReadTable should be called based on the boolean returned by the CompressXX call.
• Accessing Tables: If you need to store tables separately from the data, you can access them via the OutData and OutTable fields on the Scratch object.

The s2sx tool creates self-extracting archives with no dependencies. It can target different operating systems and architectures.

Key Features:

• Cross-platform: Use -os and -arch to specify targets (e.g., windows, linux, darwin).
• Large Archives: If the output executable exceeds the -max size (default 1GB), a .more file is created. This file must be kept alongside the executable for extraction.
• TAR Support: Use the -untar flag to make the archive automatically untar its contents upon extraction.

Supported Platforms: darwin-amd64, darwin-arm64, linux-amd64, linux-arm, linux-arm64, linux-mips64, linux-ppc64le, windows-386, windows-amd64.

S2 supports dictionary compression, which is particularly effective for small, highly repetitive data samples. A dictionary provides a lookup table at the beginning of blocks.

Important Notes:

• The same dictionary must be used for both encoding and decoding. Using the wrong dictionary will not typically trigger an error but will result in incorrect decompression.
• Blocks encoded without dictionaries can be seamlessly decompressed with a dictionary.
• S2 limits dictionary usage to the first 64KB of a block to prevent speed impacts on larger blocks.
• Dictionaries are most effective for the first few KB of data.