lz4-rs: Rust LZ4 Bindings

repository·master·Indexed 41 days ago

https://github.com/bozaro/lz4-rs

Low-level FFI bindings to the LZ4 C library for Rust, providing support for simple block compression and the LZ4 Frame API for streaming. Version 1.23.1 includes high-level wrappers like EncoderBuilder and Decoder, as well as direct access to LZ4F functions for fine-grained control over block size, checksums, and compression levels.

Tokens
5.5K
Snippets
21
Records
28
Agent score
51%

What's inside lz4-rs

  1. Use the lz4 crate for compression and decompression

    master

    The lz4 crate provides high-level wrappers around the LZ4 compression algorithm. It exposes Encoder for compression and Decoder for decompression. You can use EncoderBuilder to configure compression settings before creating an encoder.

    use lz4::Encoder;
    use lz4::Decoder;
    
    // Example usage pattern (conceptual):
    // let mut encoder = Encoder::new(writer, block_size, block_mode, content_checksum)?;
    // let mut decoder = Decoder::new(reader)?;
  2. Configure LZ4 compression modes

    master

    The CompressionMode enum allows you to select the compression strategy:

    • HIGHCOMPRESSION(i32): Uses high compression settings with a specified compression parameter.
    • FAST(i32): Uses fast compression with a specified acceleration parameter.
    • DEFAULT: Uses the default LZ4 compression settings.
    use lz4::block::CompressionMode;
    
    let mode = CompressionMode::HIGHCOMPRESSION(12);
    let fast_mode = CompressionMode::FAST(1);
    let default_mode = CompressionMode::DEFAULT;
  3. How size-prefixed compression works

    master

    When calling compress with prepend_size: true, the resulting Vec<u8> contains a 4-byte u32 header representing the original uncompressed size, followed by the compressed data.

    This pattern allows for simpler decompression: when calling decompress, passing None for the uncompressed_size parameter tells the function to parse this 4-byte prefix to determine how much memory to allocate for the output buffer.

    // Compression with prefix
    let compressed = compress(&data, None, true).unwrap();
    
    // Decompression without needing to know the size beforehand
    let original = decompress(&compressed, None).unwrap();
  4. Use LZ4 Frame API for Streaming Decompression

    master

    To decompress LZ4 frames, use the LZ4F_decompressionContext.

    Workflow:

    1. Create a context using LZ4F_createDecompressionContext with LZ4F_VERSION.
    2. (Optional) Use LZ4F_getFrameInfo to decode header information (like block size) without starting decompression.
    3. Decompress data using LZ4F_decompress. This function returns a hint of how many bytes are expected in the next call. When it returns 0, the frame is fully decoded.
    4. Free the context with LZ4F_freeDecompressionContext.
    // 1. Create context
    let mut ctx = LZ4FDecompressionContext(std::ptr::null_mut());
    unsafe { LZ4F_createDecompressionContext(&mut ctx, LZ4F_VERSION) };
    
    // 2. Decompress
    // dst_size and src_size are updated by the function
    let result = unsafe { LZ4F_decompress(ctx, dst_buffer, &mut dst_size, src_buffer, &mut src_size, &options) };
    
    if result == 0 { /* Frame fully decoded */ }
    
    // 3. Cleanup
    unsafe { LZ4F_freeDecompressionContext(ctx) };
  5. Use LZ4 Frame API for Streaming Compression

    master

    The Frame API is used for compressing data into a formal LZ4 frame format, which includes headers and checksums. This is suitable for streaming data or creating files.

    Workflow:

    1. Create a context using LZ4F_createCompressionContext with LZ4F_VERSION.
    2. Initialize the frame with LZ4F_compressBegin (writes the header).
    3. Feed data using LZ4F_compressUpdate (can be called repeatedly).
    4. Optionally use LZ4F_flush to compress buffered data without finishing the frame.
    5. Finalize the frame with LZ4F_compressEnd (writes end mark and checksum).
    6. Free the context with LZ4F_freeCompressionContext.
    // 1. Create context
    let mut ctx = LZ4FCompressionContext(std::ptr::null_mut());
    let res = unsafe { LZ4F_createCompressionContext(&mut ctx, LZ4F_VERSION) };
    
    // 2. Begin frame (writes header to dst_buffer)
    let header_size = unsafe { LZ4F_compressBegin(ctx, dst_buffer, dst_max_size, &preferences) };
    
    // 3. Update (compress data blocks)
    let bytes_written = unsafe { LZ4F_compressUpdate(ctx, dst_buffer, dst_max_size, src_buffer, src_size, &options) };
    
    // 4. End frame (finalizes frame, flushes, and writes end mark)
    let end_size = unsafe { LZ4F_compressEnd(ctx, dst_buffer, dst_max_size, &options) };
    
    // 5. Cleanup
    unsafe { LZ4F_freeCompressionContext(ctx) };
  6. Compress and decompress data with lz4

    master

    You can perform lossless compression using EncoderBuilder and decompression using Decoder.

    Compression

    Use EncoderBuilder::new() to configure compression settings (such as compression level) before calling .build(destination) to create an encoder. After writing data to the encoder, call .finish() to finalize the compression process and retrieve the result.

    Decompression

    Use Decoder::new(source) to create a decoder from a source (such as a File). The decoder implements Read, allowing you to use standard I/O utilities like io::copy to decompress data into a destination.

    extern crate lz4;
    
    use std::fs::File;
    use std::io::{self, Result};
    use std::path::{Path};
    use lz4::{Decoder, EncoderBuilder};
    
    fn compress(source: &Path, destination: &Path) -> Result<()> {
        let mut input_file = File::open(source)?;
        let output_file = File::create(destination)?;
        
        // Build an encoder with a specific compression level
        let mut encoder = EncoderBuilder::new()
            .level(4)
            .build(output_file)?;
        
        io::copy(&mut input_file, &mut encoder)?;
        
        // Finalize the compression
        let (_output, result) = encoder.finish();
        result
    }
    
    fn decompress(source: &Path, destination: &Path) -> Result<()> {
        let input_file = File::open(source)?;
        
        // Create a decoder from the input file
        let mut decoder = Decoder::new(input_file)?;
        let mut output_file = File::create(destination)?;
        
        io::copy(&mut decoder, &mut output_file)?;
    
        Ok(())
    }
  7. Configure LZ4F Compression Preferences

    master

    The LZ4FPreferences struct allows you to tune the compression behavior of the Frame API.

    FieldTypeDescription
    frame_info.block_size_idBlockSizeControls the size of compressed blocks (e.g., BlockSize::Default, BlockSize::Max4MB).
    frame_info.block_modeBlockModeLinked (default) or Independent.
    frame_info.content_checksum_flagContentChecksumNoChecksum or ChecksumEnabled.
    compression_levelc_uint0 is default (fast). Values above 16 are treated as 16.
    auto_flushc_uint1 to always flush, reducing the need for temporary buffers.
  8. Decompress LZ4 data with Decoder

    master

    The Decoder<R> struct provides a way to decompress LZ4 data streams. It implements the std::io::Read trait, allowing it to be used with any standard Rust reading utilities (like read_to_end).

    To use it, wrap an existing reader R (which must implement std::io::Read) in a new Decoder instance using Decoder::new(r).

    use lz4::Decoder;
    use std::io::{Read, Cursor};
    
    let compressed_data = vec![/* ... LZ4 compressed bytes ... */];
    let mut decoder = Decoder::new(Cursor::new(compressed_data)).unwrap();
    
    let mut decompressed_data = Vec::new();
    decoder.read_to_end(&mut decompressed_data).unwrap();
  9. Access the underlying reader from a Decoder

    master

    If you need an immutable reference to the underlying reader R without consuming the Decoder, you can use the reader() method.

    let decoder = Decoder::new(my_reader).unwrap();
    let underlying_reader = decoder.reader();
  10. Use LZ4 Block Compression (Simple API)

    master

    For simple, one-shot compression and decompression of data blocks without frame headers, use the following functions. These are ideal for small buffers where the overhead of a full frame is not desired.

    // Compress using default settings
    let compressed_size = unsafe { LZ4_compress_default(src_ptr, dst_ptr, src_size, max_dst_size) };
    
    // Compress with fast mode and acceleration
    let compressed_size = unsafe { LZ4_compress_fast(src_ptr, dst_ptr, src_size, max_dst_size, acceleration) };
    
    // Compress with High Compression (HC) level
    let compressed_size = unsafe { LZ4_compress_HC(src_ptr, dst_ptr, src_size, dst_capacity, compression_level) };
    
    // Safe decompression
    let decompressed_size = unsafe { LZ4_decompress_safe(src_ptr, dst_ptr, compressed_size, max_dst_size) };