gzp

gzp is a Rust crate providing multi-threaded encoding and decoding. It acts as a near drop-in replacement for std::io::Write by compressing chunks of data in parallel and writing them to an underlying writer in the original byte order. This enables significantly faster compression for large datasets.

Supported encodings include:

• Gzip (via flate2)
• Zlib (via flate2)
• Raw Deflate (via flate2)
• Snappy (via rust-snappy)
• BGZF (block compression format limited to 64 Kb blocks)
• Mgzip (block compression format with no block size limit)

By default, gzp enables deflate_default and libdeflate features, which use the best performing zlib implementation and libdeflater for block gzip formats. You can customize your dependency based on your requirements:

• Default (High performance): Uses libdeflate and flate2.
• Rust-only backend: Disables Zlib support by using deflate_rust.
• Snappy only: Disables all other encodings to use only Snappy.

# Default (Deflate default)
[dependencies]
gzp = { version = "*" }

# Rust backend (Zlib will not be available)
[dependencies]
gzp = { version = "*", default-features = false, features = ["deflate_rust"] }

# Snappy only
[dependencies]
gzp = { version = "*", default-features = false, features = ["snap_default"] }

The Snap struct implements the FormatSpec and SyncWriter traits, allowing it to be used within the gzp framework for Snappy compression.

Key characteristics:

• Format: Uses the FrameEncoder format where each block is a frame.
• Compression Level: The Compression level is ignored for the Snap format as Snappy does not use levels in this implementation.
• Dictionary Support: needs_dict() always returns false, meaning dictionaries are not supported for this format.
• Parallelization: Can be used with ParCompressBuilder for multi-threaded compression or SyncZBuilder for synchronous compression.

The gzp crate provides implementations for several DEFLATE-based compression formats:

• Gzip: Standard Gzip stream with gzip header and footer (uses Crc32 checksum).
• Zlib: Zlib stream with zlib header and footer (uses Adler32 checksum). Requires any_zlib feature.
• Raw Deflate: Contiguous raw deflate stream without headers or footers.
• Mgzip: A multi-member Gzip format.
• Bgzf: Block Gzip format (BGZF), often used in bioinformatics.

The gzp crate supports the following encodings:

• Gzip
• Zlib
• BGZF
• Mgzip
• Raw Deflate
• Snap Frame Encoding

The ZBuilder provides a unified API to create either a parallel or a synchronous compression writer. If num_threads is greater than 1, it returns a parallel compressor (similar to pigz). If num_threads is 0 or 1, it returns a synchronous writer. This allows you to write code that works regardless of the available thread count.

# #[cfg(feature = "deflate")] {
use std::io::Write;
use gzp::{deflate::Gzip, ZBuilder, ZWriter};

let mut writer = vec![];
let mut parz = ZBuilder::<Gzip, _>::new()
    .num_threads(0)
    .from_writer(writer);
parz.write_all(b"This is a first test line\n").unwrap();
parz.write_all(b"This is a second test line\n").unwrap();
parz.finish().unwrap();
# }

The ZBuilder provides a way to create a parallel writer. It returns a trait object that is transparent over ParZ or SyncZ. You can specify the number of threads using .num_threads(), where 0 typically indicates using all available threads.

Example using Gzip encoding:

use std::{env, fs::File, io::Write};

use gzp::{deflate::Gzip, ZBuilder, ZWriter};

fn main() {
    let mut writer = vec![];
    // ZBuilder will return a trait object that transparent over `ParZ` or `SyncZ`
    let mut parz = ZBuilder::<Gzip, _>::new()
        .num_threads(0)
        .from_writer(writer);
    parz.write_all(b"This is a first test line\n").unwrap();
    parz.write_all(b"This is a second test line\n").unwrap();
    parz.finish().unwrap();
}

For Snappy encoding, use ParZBuilder to create a ParZ<Snap> writer. This follows a similar chunked processing pattern as the Mgzip example.

use gzp::{parz::{ParZ, ParZBuilder}, snap::Snap};
use std::io::{Read, Write};

fn main() {
    let chunksize = 64 * (1 << 10) * 2;

    let stdout = std::io::stdout();
    let mut writer: ParZ<Snap> = ParZBuilder::new().from_writer(stdout);

    let stdin = std::io::stdin();
    let mut stdin = stdin.lock();

    let mut buffer = Vec::with_capacity(chunksize);
    loop {
        let mut limit = (&mut stdin).take(chunksize as u64);
        limit.read_to_end(&mut buffer).unwrap();
        if buffer.is_empty() {
            break;
        }
        writer.write_all(&buffer).unwrap();
        buffer.clear();
    }
    writer.finish().unwrap();
}

Errors occurring in background decompression threads are propagated to the main thread when calling .read() or .finish().

Common error types related to decompression include:

• GzpError::BufferSize: Returned if the requested buffer size is smaller than DICT_SIZE.
• GzpError::NumThreads: Returned if num_threads() is called with 0.
• GzpError::InvalidCheck: Returned if the checksum of a decompressed block does not match the expected value in the footer.
• GzpError::Io: Wraps standard std::io::Error instances.

For synchronous (single-threaded) compression, use SyncZBuilder. This is useful when you want to specify a compression level and initialize a writer from an existing std::io::Write implementation.

use std::io::Write;
use gzp::{deflate::Gzip, syncz::SyncZBuilder, ZWriter};

let mut z = SyncZBuilder::<Gzip, _>::new()
    .compression_level(Compression::new(6))
    .from_writer(out_writer);

z.write_all(input).unwrap();
z.finish().unwrap();