Aircompressor

Aircompressor supports several algorithms depending on your performance and resource requirements:

• Zstandard (Zstd): Recommended for most use cases. Provides superior compression and performance compared to zlib. Use ZstdNativeCompressor/Decompressor (native) or ZstdJavaCompressor/Decompressor (Java).
• LZ4: Extremely fast; ideal for high-performance requirements. Use Lz4NativeCompressor/Decompressor or Lz4JavaCompressor/Decompressor. Supports the LZ4 frame format via Lz4FrameNativeCompressor or Lz4FrameJavaCompressor.
• Snappy: Good for extremely resource-limited environments due to guaranteed memory usage. Use SnappyNativeCompressor/Decompressor or SnappyJavaCompressor/Decompressor.
• LZO: Provided for compatibility only. Use LzoCompressor/Decompressor (Java implementation only).
• Deflate: Provided for compatibility with gzip/zlib. Use DeflateCompressor/Decompressor.

CBZip2OutputStream is an OutputStream that compresses data into the BZip2 format (excluding the file header characters).

Important Requirements:

• Magic Bytes: The caller is responsible for writing the two BZip2 magic bytes "BZ" to the destination stream before initializing this class.
• Memory Management: Compression requires significant memory. You should call .close() or .finish() as soon as possible to release allocated memory.
• Thread Safety: Instances of this class are not thread-safe.

Block Size and Memory Usage: You can tune the blockSize (expressed in 100k units, from 1 to 9) to balance compression ratio, speed, and memory usage. A lower block size reduces memory usage and can increase speed but may lower the compression ratio.

Memory Estimation Formulas:

• Compression memory usage: 400k + (9 * blocksize)
• Decompression memory usage: 65k + (5 * blocksize)

// Example setup (pseudo-code for context)
OutputStream fileOut = new FileOutputStream("data.bz2");
fileOut.write('B');
fileOut.write('Z');

try (CBZip2OutputStream bzipOut = new CBZip2OutputStream(fileOut)) {
    bzipOut.write(myData);
}

LzopHadoopOutputStream is a Hadoop-compatible output stream that compresses data using the LZO algorithm with the lzop file format. It is designed to be used as a wrapper around an existing OutputStream.

Constructor

LzopHadoopOutputStream(OutputStream out, int bufferSize)

• out: The underlying destination OutputStream.
• bufferSize: The size of the internal input buffer. A larger buffer can improve compression efficiency but increases memory usage.

Lifecycle Methods

• write(int b): Writes a single byte to the stream.
• write(byte[] buffer, int offset, int length): Writes a portion of a byte array to the stream. This method is optimized to avoid extra copies when possible.
• finish(): Flushes any remaining data in the input buffer by compressing and writing the final chunk.
• flush(): Flushes the underlying OutputStream.
• close(): Finalizes the LZO stream by calling finish(), writing a termination marker (0), and closing the underlying OutputStream.

OutputStream out = new FileOutputStream("data.lzo");
try (LzopHadoopOutputStream lzoOut = new LzopHadoopOutputStream(out, 65536)) {
    lzoOut.write("Hello, LZO!".getBytes());
}

CBZip2InputStream is an InputStream that decompresses data in the BZip2 format.

Important Usage Notes

• Header Requirement: The constructor expects the stream to start at the first byte after the BZip2 magic bytes (Bz). You must manually skip the first two bytes of the BZip2 stream before passing the InputStream to the constructor, otherwise an exception will be thrown.
• Memory Management: Decompression requires large amounts of memory. You should call .close() as soon as possible to force the release of allocated memory.
• Performance: The implementation reads from the source stream using single-byte read() calls. It is highly recommended to wrap your source InputStream in a BufferedInputStream to improve performance.
• Thread Safety: Instances of this class are not thread-safe.
• Block-based Reading: This implementation supports a mode where it identifies BZip2 block boundaries. In this mode, it can report the position in the compressed stream at the end of each block.

// Assuming 'compressedStream' is a BZip2 stream
// 1. Skip the first two bytes ('B' and 'z')
InputStream bufferedStream = new BufferedInputStream(compressedStream);
bufferedStream.skip(2);

// 2. Initialize the decompressor
try (CBZip2InputStream bzip2Stream = new CBZip2InputStream(bufferedStream)) {
    int byteRead;
    while ((byteRead = bzip2Stream.read()) != -1) {
        // Process decompressed byte
    }
}