lz4 Go Library

The lz4c command-line interface tool allows you to compress and decompress LZ4 files from your terminal. You can install it using go install.

go install github.com/pierrec/lz4/v4/cmd/lz4c@latest

To use the LZ4 compression library in your Go projects, use the go get command to fetch the v4 package.

go get github.com/pierrec/lz4/v4

The lz4 package uses a functional options pattern to configure Writer, Reader, and CompressingReader instances. You can pass one or more Option functions to the constructor of these types to customize compression behavior, block sizes, checksums, and concurrency.

Commonly used options include:

• BlockSizeOption(size): Sets the maximum size of compressed blocks.
• CompressionLevelOption(level): Sets the compression intensity.
• ConcurrencyOption(n): Sets the number of goroutines used for compression.
• ChecksumOption(flag): Enables/disables content checksums.

// Example of applying options (assuming Writer is available)
writer := lz4.NewWriter(output, 
    lz4.BlockSizeOption(lz4.Block1Mb), 
    lz4.CompressionLevelOption(lz4.Level3),
    lz4.ConcurrencyOption(4),
) 

The lz4 package provides a streaming interface. You can use lz4.NewWriter to compress data into an io.Writer and lz4.NewReader to decompress data from an io.Reader. When using a pipe for streaming, ensure you close the writer and the pipe to properly terminate the stream.

// Compress and uncompress an input string.
s := "hello world"
r := strings.NewReader(s)

// The pipe will uncompress the data from the writer.
pr, pw := io.Pipe()
zw := lz4.NewWriter(pw)
zr := lz4.NewReader(pr)

go func() {
	// Compress the input string.
	_, _ = io.Copy(zw, r)
	_ = zw.Close() // Make sure the writer is closed
	_ = pw.Close() // Terminate the pipe
}()

_, _ = io.Copy(os.Stdout, zr)

// Output:
// hello world

The following flags are available for the lz4c command-line tool:

Global:
  -version        print the program version

Compress subcommand:
  -bc            enable block checksum
  -l int         compression level (0=fastest)
  -sc            disable stream checksum
  -size string   block max size [64K,256K,1M,4M] (default "4M")

Uncompress subcommand:
  (No specific flags listed in help)

When using the -bench flag with the uncompress command, the tool performs the following steps for each iteration:

1. Reads the entire compressed file into memory.
2. Triggers a Garbage Collection (runtime.GC()) to ensure clean measurements.
3. Measures the time taken to decompress the data into an io.Discard writer.
4. Calculates and prints:
   • Compressed size vs Uncompressed size
   • Compression percentage
   • Elapsed time (rounded to milliseconds)
   • Throughput in MB/s

# Example command to run 5 benchmark iterations
$ lz4c uncompress -bench 5 my_archive.lz4

# Example output format:
# Reading my_archive.lz4...
# Decompressing...
# 1024 -> 5120 [500%]; 12ms, 426.67MB/s

The lz4c tool provides subcommands for compression and decompression. It supports reading from files or from stdin and writing to stdout.

# Compress files
lz4c compress [arguments] [<file name> ...]

# Uncompress files
lz4c uncompress [arguments] [<file name> ...]

Use CompressionLevelOption to define the compression intensity. Higher levels provide better compression ratios but are slower.

Available CompressionLevel constants:

• Fast (Default)
• Level1 through Level9

// Use a high compression level
opt := lz4.CompressionLevelOption(lz4.Level9)

Call Flush() to immediately compress and write any pending data currently held in the Writer's internal buffer to the underlying destination. This ensures that even if a block is not yet full, the data is processed.

err := zw.Flush()

Use LegacyOption(legacy bool) to enable support for writing LZ4 frames in the legacy format. This is required for compatibility with certain older implementations or specific use cases like compressed Linux kernel images.

// Enable legacy frame format for a Writer
opt := lz4.LegacyOption(true)

The Apply(options ...Option) method allows you to change the configuration of an existing Writer. This is useful for setting compression levels, concurrency, or block sizes after initialization. Note that Apply will call Reset internally, so it should be used before writing data. If the writer is already in an error state or has been closed, Apply will return an error.

// Example of applying options to a writer
err := zw.Apply(
    lz4.DefaultBlockSizeOption,
    lz4.DefaultConcurrency,
)

To avoid buffer overflow or failed compression, use CompressBlockBound(n) to calculate the maximum size a buffer of size n might require when it is not compressible. This is the safe size to allocate for the destination buffer.

import "github.com/pierrec/lz4/v4"

srcSize := 1024
maxDstSize := lz4.CompressBlockBound(srcSize)
dst := make([]byte, maxDstSize)