DOCSEARCHbyAlgolia
Submit Docs

NativeCompressions

repository·main·Indexed 43 days ago

https://github.com/cysharp/nativecompressions

High-performance native bindings for LZ4, Zstandard, and OpenZL optimized for .NET. Features support for Span<T>, PipeReader/Writer, and async streaming, providing both high-level static methods and low-level encoders/decoders for non-allocating compression.

Tokens
4.4K
Snippets
12
Records
18
Agent score
55%

What's inside NativeCompressions

  1. Overview of System.IO.Pipelines

    main

    System.IO.Pipelines is a library designed for high-performance I/O by simplifying the parsing of streaming data. It aims to reduce the boilerplate and complex code flows typically associated with managing specialized I/O operations.

    Key Features:

    • Buffer Management: Provides single producer/single consumer byte buffer management.
    • Complexity Reduction: Minimizes the boilerplate code required for high-performance streaming data parsing.

  3. Core types in System.IO.Pipelines

    main

    The library is built around three primary types that manage the flow of data through a pipeline:

    • System.IO.Pipelines.Pipe: The central object that manages the connection between a reader and a writer.
    • System.IO.Pipelines.PipeWriter: Used by the producer to write data into the pipeline.
    • System.IO.Pipelines.PipeReader: Used by the consumer to read data from the pipeline.

  4. Zstandard API Overview

    main

    The Zstandard API is similar to LZ4. It provides static methods in the Zstandard class and streamless-streaming APIs via ZstandardEncoder and ZstandardDecoder.

    Key Differences from LZ4:

    • ZstandardEncoder and ZstandardDecoder are sealed classes (not structs) that own a single native context. They are safe to cache and share by reference (e.g., in a serializer).
    • They are not thread-safe.
    • Always call Dispose(). Using an instance after disposal throws ObjectDisposedException.
    • The Compress method returns an OperationStatus (containing bytesConsumed, bytesWritten, and isFinalBlock), matching the BrotliEncoder pattern.

  5. Install NativeCompressions via NuGet

    main

    To use NativeCompressions in a standard .NET project, add the package using the dotnet CLI:

    dotnet add package NativeCompressions

    Note: This library is currently in preview. The API may change, and it is not recommended for production environments.

  6. Install NativeCompressions in Unity

    main

    To use NativeCompressions in Unity:

    1. Install via NuGetForUnity: Search for NativeCompressions and install.
    2. Unsupported Runtimes: If you need win-arm64, linux-arm64, android-arm, android-x64, or ios-x64, replace ProjectSettings/Packages/com.github-glitchenzo.nugetforunity/NativeRuntimeSettings.json with the official workaround version.
    3. iOS Support: iOS requires static linking. Install the NativeCompressions.Unity editor package via the Package Manager using this URL: https://github.com/Cysharp/NativeCompressions.git?path=src/NativeCompressions.Unity

  7. How to use System.Threading.Channels for async data passing

    main

    The System.Threading.Channels library provides abstractions for passing data asynchronously between producers and consumers. You can create channels using factory methods (like Channel.CreateUnbounded<T>()) and interact with them via a Writer to publish data and a Reader to consume data. Consumption can be performed asynchronously using ReadAllAsync() within an await foreach loop.

    using System;
using System.Threading.Channels;
using System.Threading.Tasks;

Channel<int> channel = Channel.CreateUnbounded<int>();

Task producer = Task.Run(async () =>
{
    int i = 0;
    while (true)
    {
        channel.Writer.TryWrite(i++);
        await Task.Delay(TimeSpan.FromSeconds(1));
    }
});

Task consumer = Task.Run(async () =>
{
    await foreach (int value in channel.Reader.ReadAllAsync())
    {
        Console.WriteLine(value);
    }
});

await Task.WhenAll(producer, consumer);

  8. Simple Compression with LZ4 and Zstandard

    main

    For quick, one-shot compression and decompression of byte arrays, use the static LZ4.Compress/Decompress or Zstandard.Compress/Decompress methods.

    // for LZ4
using NativeCompressions;
byte[] compressed = LZ4.Compress(sourceData);
byte[] decompressed = LZ4.Decompress(compressed);

// for Zstandard
using NativeCompressions;
byte[] compressed = Zstandard.Compress(sourceData);
byte[] decompressed = Zstandard.Decompress(compressed);

  9. Low-level Streaming LZ4 Decompression with LZ4Decoder

    main

    Use LZ4Decoder to decompress data in a streamless manner. The Decompress method returns an OperationStatus and handles incomplete data.

    OperationStatus values:

    • Done: All data has been restored.
    • NeedMoreData: More input data is required.
    • DestinationTooSmall: The destination buffer is insufficient.
    // while(status != OperationStatus.Done && source.Length > 0)
OperationStatus status = decoder.Decompress(source, destination, out int bytesConsumed, out int bytesWritten);
source = source.Slice(bytesConsumed);
destination = destination.Slice(bytesWritten);

  10. High-level Async Streaming for LZ4

    main

    Use LZ4.CompressAsync and LZ4.DecompressAsync to stream data from/to ReadOnlyMemory<byte>, ReadOnlySequence<byte>, SafeFileHandle, Stream, or PipeReader to PipeWriter.

    Parallelism:

    • You can specify int? maxDegreeOfParallelism.
    • If $\ge 2$ and the source is $\ge 1$MB, parallel processing occurs.
    • Warning: Parallel compression cannot produce a content checksum. Combining it with ContentChecksumFlag throws NotSupportedException.
    • ASP.NET Recommendation: Always set maxDegreeOfParallelism: 1 to avoid reducing overall server throughput.
    // Parallel Compression from File to File
using SafeFileHandle sourceHandle = File.OpenHandle("foo.bin");
using var dest = new FileStream("foo.lz4", FileMode.OpenOrCreate, FileAccess.Write, FileShare.None, bufferSize: 1, useAsync: true);
await LZ4.CompressAsync(sourceHandle, PipeWriter.Create(dest), maxDegreeOfParallelism: Environment.ProcessorCount);

  11. Experimental OpenZL Bindings

    main

    OpenZL is a new compression library from Facebook. NativeCompressions provides experimental C bindings via NativeCompressions.Interop.OpenZLNativeMethods. These are not part of the 1.0 release and are not available via NuGet. To use them, you must build src/NativeCompressions.OpenZL.csproj from source.

    using NativeCompressions.Interop;
using static NativeCompressions.Interop.OpenZLNativeMethods;

public static unsafe int Compress(ReadOnlySpan<byte> source, Span<byte> destination)
{
    fixed (byte* src = source)
    fixed (byte* dest = destination)
    {
        var cctx = ZL_CCtx_create();
        try
        {
            var cgraph = ZL_Compressor_create();
            try
            {
                const int ZSTRONG_EXAMPLE_FORMAT_VERSION = 16;
                ThrowIfError(ZL_Compressor_setParameter(cgraph, ZL_CParam.ZL_CParam_formatVersion, ZSTRONG_EXAMPLE_FORMAT_VERSION));
                ThrowIfError(ZL_Compressor_selectStartingGraphID(cgraph, new ZL_GraphID { gid = (uint)ZL_StandardGraphID.ZL_StandardGraphID_zstd }));
                ThrowIfError(ZL_CCtx_refCompressor(cctx, cgraph));

                var written = ZL_CCtx_compress(cctx, dest, (nuint)destination.Length, src, (nuint)source.Length);
                ThrowIfError(written);

                return (int)written._value._value;
            }
            finally
            {
                ZL_Compressor_free(cgraph);
            }
        }
            finally
            {
                ZL_CCtx_free(cctx);
            }
    }
}

  12. Raw LZ4 Block Compression

    main

    Use LZ4.Block for raw LZ4 block compression without the frame format overhead.

    // Get max compressed size
var maxSize = LZ4.Block.GetMaxCompressedLength(source.Length);

// Compress block
var destination = new byte[maxSize];
var compressedSize = LZ4.Block.Compress(source, destination);
var compressed = destination.AsSpan(0, compressedSize).ToArray();

// Decompress block
destination = new byte[source.Length];
var decompressedSize = LZ4.Block.Decompress(compressed, destination);
var decompressed = destination.AsSpan(0, decompressedSize).ToArray();