DOCSEARCHbyAlgolia
Submit Docs

SharpCompress Documentation

repository·master·Indexed 25 days ago

https://github.com/adamhathcock/sharpcompress

A pure C# compression library for .NET supporting a wide range of archive formats. It is optimized for processing large files via non-seekable streams and provides both forward-only and random access APIs. Supported runtimes include .NET Framework 4.8, .NET Standard 2.0/2.1, and .NET 6.0, 8.0, and 10.0. The library supports unpacking for formats such as unrar, un7zip, unzip, and untar, and compression for zip, tar, bzip2, gzip, lzip, zstandard, and 7zip.

Tokens
31K
Snippets
70
Records
120
Agent score
83%

What's inside SharpCompress

  1. Overview of SharpCompress

    master

    SharpCompress is a pure C# compression library for .NET that supports a wide variety of archive formats. It is designed to handle both forward-only reading (ideal for processing large files from non-seekable streams like network downloads) and file random access.

    Supported Formats:

    • Unpacking: unrar, un7zip, unzip, untar, unbzip2, ungzip, unlzip, unxz, unzstd, unarc, unarj, unace, and unlzw.
    • Compression: zip, tar, bzip2, gzip, lzip, zstandard, and 7zip.

    Supported Runtimes:

    • .NET Framework 4.8
    • .NET Standard 2.0/2.1
    • .NET 6.0, .NET 8.0, and .NET 10.0

    All I/O operations support async/await for improved performance and scalability.

  2. Identify the main Tar API entry points

    master

    SharpCompress provides four primary ways to interact with Tar archives, depending on whether you need to stream data, manipulate an existing archive, or detect formats:

    • TarFactory: The central entry point for format detection and creating readers or writers.
    • TarArchive: An archive-level API used for enumerating and rewriting (modifying) existing tar archives.
    • TarReader: A forward-only, streaming API designed for efficient, one-pass extraction of tar entries.
    • TarWriter: A forward-only API for creating new tar archives.

    Use TarWriterOptions to configure output compression, stream ownership, archive finalization, encoding, and the header write format.

  3. Implement a CustomDecoder for mixed or complex encodings

    master

    If a single encoding is insufficient (e.g., an archive contains mixed encodings), you can provide a CustomDecoder delegate within ArchiveEncoding. This delegate receives the raw byte data, an offset, and a length, and must return the decoded string.

    You can implement logic to attempt multiple encodings (like UTF-8 followed by a fallback) within this delegate.

    var options = new ReaderOptions
{
    ArchiveEncoding = new ArchiveEncoding
    {
        CustomDecoder = (data, offset, length) =>
        {
            var bytes = new byte[length];
            Array.Copy(data, offset, bytes, 0, length);
            
            try
            {
                return Encoding.UTF8.GetString(bytes);
            }
            catch
            {
                return Encoding.GetEncoding(932).GetString(bytes);
            }
        }
    }
};

using (var archive = ZipArchive.OpenArchive("mixed.zip", options))
{
    foreach (var entry in archive.Entries)
    {
        Console.WriteLine(entry.Key);
    }
}

  4. Understand the difference between ArchiveType and CompressionType

    master

    SharpCompress distinguishes between high-level archive containers and pure compression algorithms:

    • ArchiveType: Represents archive containers exposed by high-level APIs. Examples include Rar, Zip, Tar, SevenZip, GZip, Arc, Arj, Ace, and Lzw.
    • CompressionType: Represents pure compression algorithms or single-file wrappers. XZ and ZStandard are represented as CompressionType values rather than ArchiveType values.

    Key distinction: Formats like Zip and 7Zip are archive formats that use compression methods (like DEFLATE or LZMA) to store data.

  5. Implement a custom compression provider

    master

    To extend SharpCompress with your own compression logic, you can:

    • Implement ICompressionProvider directly.
    • Derive from CompressionProviderBase to get default async method implementations.
    • Derive from DecompressonOnlyProviderBase for read-only codecs.

    Providers can access CompressionContext to retrieve information about stream size, seekability, reader options, compression properties, and format-specific metadata. For formats requiring pre- or post-stream data (like LZMA or PPMd), implement ICompressionProviderHooks to supply necessary header bytes.

  6. Choose the correct API for accessing archives

    master

    SharpCompress provides three distinct API patterns for interacting with compressed data. Choosing the right one depends on whether you need random access or sequential processing:

    • Archive classes: Use these for random access to a seekable stream. These allow you to jump to specific entries within the archive.
    • Reader classes: Use these for forward-only reading on a stream. These are suitable for streaming data where you process entries one after another without jumping back.
    • Writer classes: Use these for forward-only writing on a stream.

  7. How Tar detection works

    master

    Tar detection is performed by TarFactory using a content-probing mechanism:

    1. The incoming stream is wrapped in a SharpCompressStream.
    2. A rewind buffer (sized by TarWrapper.MaximumRewindBufferSize) is used to allow probing.
    3. The system probes registered wrappers in order.
    4. If a wrapper matches, a decompression stream is created.
    5. TarArchive.IsTarFile or TarArchive.IsTarFileAsync is called on the decompressed stream to verify it is a valid tar payload.

    Key Implications:

    • Detection is content-based, not extension-based.
    • A matching wrapper is not enough; the payload must also be a valid tar file.
    • TarArchive only detects raw tar. For compressed wrappers (e.g., .tar.gz), use TarReader.

  8. How sync methods are generated from async methods

    master

    SharpCompress uses the Zomp.SyncMethodGenerator to eliminate code duplication. Instead of hand-maintaining separate sync and async versions of a method (e.g., Foo.cs and Foo.Async.cs), the library now maintains only the async version and uses a source generator to produce the sync version automatically.

    The Mapping Logic: When generating a sync version from an async method, the following transformations occur:

    • The Async suffix is dropped from the method name.
    • Task/ValueTask $\rightarrow$ void (or Task<T>/ValueTask<T> $\rightarrow$ T).
    • CancellationToken and IProgress<T> are dropped (unless PreserveCancellationToken or PreserveProgress are specified).
    • Memory<T> $\rightarrow$ Span<T>.
    • ReadOnlyMemory<T> $\rightarrow$ ReadOnlySpan<T>.
    • Modifiers like public, override, virtual, static, and sealed are copied verbatim.
    • Extension method invocations (e.g., this.Skip()) are converted to static calls (e.g., StreamExtensions.Skip(this)).

  9. Understand Tar Metadata Surface

    master

    The metadata available to you depends on whether you are reading from an archive or writing to one.

    Reading Metadata

    When reading entries, the TarEntry object surfaces the following metadata:

    • name
    • link target
    • mode
    • uid
    • gid
    • size
    • last modified time

    Writing Metadata

    The writer has a narrower metadata surface. When writing, you can set:

    • LastModifiedTime
    • Name
    • Size
    • Entry type (specifically for File or Directory)

    Note: The current writer uses fixed defaults for mode, owner id, and group id rather than performing a full metadata round-trip.

  10. Performance considerations for SharpCompress

    master

    Optimize your implementation using the following guidelines:

    Memory Efficiency

    • Avoid loading entire archives in memory: Use the Reader API for large files to stream data.
    • Process entries sequentially: This is especially important for solid archives.
    • Use appropriate buffer sizes: Use larger buffers when performing network I/O.
    • Dispose streams promptly: Always free resources as soon as they are no longer needed.

    Algorithm Selection

    • Archive API: Best for small archives where random access is required.
    • Reader API: Most efficient for large files or streaming scenarios.
    • Solid archives: Extraction is significantly faster when processed sequentially.
    • Compression levels: Balance the trade-off between processing speed and resulting file size.

  11. Limitations of the Tar writer API

    master

    The current public Tar writer API has several functional limitations that developers should be aware of:

    • No Link Support: While the reader can handle symbolic and hard links, the writer API currently only supports creating regular files and directories. You cannot create symlinks or hardlinks in a new archive using the writer.
    • Metadata Fidelity: The writer does not perform full metadata round-tripping. It uses fixed defaults for certain fields like mode, uid, and gid, meaning newly created archives may not preserve the exact metadata of the original source files.
    • Sparse Files: Sparse files are not semantically implemented. Sparse entries may be treated as ordinary entries rather than sparse files with holes.

  12. Choose between Archive API and Reader API

    master

    Select an API based on your archive size, stream type, and access requirements:

    • Archive API: Best for small-to-medium archives where you need random access to specific entries. It requires a seekable stream (like a FileStream or MemoryStream) and loads all entries into memory.
    • Reader API: Best for large archives (>100 MB), memory-constrained environments, or non-seekable streams (like network streams or pipes). It processes one entry at a time with a minimal memory footprint but only supports forward-only (sequential) access.
    // Archive API: Fast for random access (requires seekable stream)
using (var archive = ZipArchive.OpenArchive("archive.zip"))
{
    var specific = archive.Entries.FirstOrDefault(e => e.Key == "file.txt");
    specific?.WriteToFile(@"C:\output\file.txt");
}

// Reader API: Best for large files or streaming (works with non-seekable streams)
using (var stream = File.OpenRead("large.zip"))
using (var reader = ReaderFactory.OpenReader(stream))
{
    while (reader.MoveToNextEntry())
    {
        reader.WriteEntryToDirectory(@"C:\output");
    }
}