DOCSEARCHbyAlgolia
Submit Docs

lz4-java

repository·master·Indexed 22 days ago

https://github.com/lz4/lz4-java

A Java implementation of the LZ4 compression algorithm and xxHash hashing algorithm. It offers high-performance compression and decompression via JNI bindings to C, pure Java, or Unsafe-optimized implementations. The library supports Fast Scan (LZ4), High Compression (LZ4 HC), and interoperable LZ4 Frame formats, as well as non-cryptographic hashing via XXHashFactory.

Tokens
1.7K
Snippets
4
Records
6
Agent score
28%

What's inside lz4-java

  1. Understand LZ4 compression methods in Java

    master

    The library provides three main compression approaches:

    1. Fast Scan (LZ4): Low memory footprint (~16 KB), very fast with skipping heuristics, and reasonable compression ratios.
    2. High Compression (LZ4 HC): Medium memory footprint (~256 KB), significantly slower (~10x slower than LZ4), but provides better compression ratios.
    3. Interoperable (LZ4 Frame): An overlay on top of LZ4 that makes the data system and library independent. Use this version to share compressed data with other systems or languages (e.g., the lz4 CLI tool).

    Note that LZ4 and LZ4 HC streams use the same compression format and can be decompressed by the same lz4-java decompressor instance.

  2. Build lz4-java from source

    master

    To build the project from source, you need JDK 7+, Ant 1.10.2+, and Ivy.

    1. Initialize Submodules:
      git submodule init
git submodule update
    2. Bootstrap Ivy: If Ivy is not installed, run:
      ant ivy-bootstrap
    3. Build: Run ant to compile the JNI bindings, Java sources (including Unsafe versions), and generate the JAR file in the dist directory.

    The resulting JAR contains the Java classes, the native library, and the JNI bindings. When added to your classpath, the native library is automatically copied to a temporary directory and dynamically linked.

    git submodule init
git submodule update
ant ivy-bootstrap
ant

  3. Download lz4-java artifacts

    master

    You can download released artifacts from Maven Central:

    • Standard artifacts (including JNI bindings): org.lz4:lz4-java
    • Pure-Java artifacts (Safe and Unsafe versions, no JNI): org.lz4:lz4-pure-java

  4. Compute hashes using xxhash Java

    master

    The library provides xxhash, a non-cryptographic, extremely fast, and high-quality hash function. You can use XXHashFactory to create streaming hash instances. For streaming, you must provide a seed (an integer) to initialize the hash value; ensure you use the same seed for both hashing and verification.

    XXHashFactory factory = XXHashFactory.fastestInstance();

byte[] data = "12345345234572".getBytes("UTF-8");
ByteArrayInputStream in = new ByteArrayInputStream(data);

int seed = 0x9747b28c; // used to initialize the hash value, use whatever
                       // value you want, but always the same
StreamingXXHash32 hash32 = factory.newStreamingHash32(seed);
byte[] buf = new byte[8]; // for real-world usage, use a larger buffer, like 8192 bytes
for (;;) {
  int read = in.read(buf);
  if (read == -1) {
    break;
  }
  hash32.update(buf, 0, read);
}
int hash = hash32.getValue();

  5. Use LZ4Frame streams for interoperable compression

    master

    For data that needs to be shared with other systems or languages, use LZ4FrameOutputStream and LZ4FrameInputStream. These implement the LZ4 Frame format, which is compatible with the lz4 CLI tool and other independent LZ4 implementations.

    byte[] data = "12345345234572".getBytes("UTF-8");
final int decompressedLength = data.length;

LZ4FrameOutputStream outStream = new LZ4FrameOutputStream(new FileOutputStream(new File("test.lz4")));
outStream.write(data);
outStream.close();

byte[] restored = new byte[decompressedLength];
LZ4FrameInputStream inStream = new LZ4FrameInputStream(new FileInputStream(new File("test.lz4")));
inStream.read(restored);
inStream.close();

  6. Compress and decompress data using LZ4Factory

    master

    To use LZ4, obtain an LZ4Factory instance (e.g., via LZ4Factory.fastestInstance()). You can then request compressors and decompressors.

    There are two ways to decompress:

    1. Fast Decompression: Use LZ4FastDecompressor when the exact decompressed length is known.
    2. Safe Decompression: Use LZ4SafeDecompressor when only the compressed length is known. This method is slightly slower and requires an over-sized destination buffer.
    LZ4Factory factory = LZ4Factory.fastestInstance();

byte[] data = "12345345234572".getBytes("UTF-8");
final int decompressedLength = data.length;

// compress data
LZ4Compressor compressor = factory.fastCompressor();
int maxCompressedLength = compressor.maxCompressedLength(decompressedLength);
byte[] compressed = new byte[maxCompressedLength];
int compressedLength = compressor.compress(data, 0, decompressedLength, compressed, 0, maxCompressedLength);

// decompress data
// - method 1: when the decompressed length is known
LZ4FastDecompressor decompressor = factory.fastDecompressor();
byte[] restored = new byte[decompressedLength];
int compressedLength2 = decompressor.decompress(compressed, 0, restored, 0, decompressedLength);
// compressedLength == compressedLength2

// - method 2: when the compressed length is known (a little slower)
// the destination buffer needs to be over-sized
LZ4SafeDecompressor decompressor2 = factory.safeDecompressor();
int decompressedLength2 = decompressor2.decompress(compressed, 0, compressedLength, restored, 0);
// decompressedLength == decompressedLength2