fastzip

repository·main·Indexed 42 days ago

https://github.com/saracen/fastzip

An opinionated, high-performance Zip archiver and extractor for Go. It features concurrent file processing, efficient buffer recycling, and strict security boundaries via chroot directories to prevent traversal attacks. It preserves permissions, Unix ownership (UID/GID), and modification times, utilizing github.com/klauspost/compress/flate by default for improved throughput.

Tokens
5.2K
Snippets
28
Records
29
Agent score
48%

What's inside fastzip

  1. Configure concurrency for Archiver and Extractor

    main

    You can tune the performance of fastzip by adjusting the concurrency levels for archiving and extraction using functional options.

    Available options:

    • WithArchiverConcurrency(n int): Sets the number of concurrent archiving operations.
    • WithExtractorConcurrency(n int): Sets the number of concurrent extraction operations.

    Benchmarks indicate that increasing concurrency (e.g., to 8 or 16) significantly improves throughput on multi-core systems when using the klauspost/compress/flate library.

  2. Create a Zip archive with Archiver

    main

    Use fastzip.NewArchiver to create an archiver instance. The archiver requires an io.Writer (like an *os.File) and a base directory path. For security, all archiving operations are restricted to occur within this specified directory.

    To archive files, you must provide a map of os.FileInfo representing the files you wish to include. You can use filepath.Walk to build this map.

    Key features:

    • Preserves permissions, ownership (uid, gid on Linux/Unix), and modification times.
    • Uses github.com/klauspost/compress/flate by default for high performance.
    • Supports concurrent file archiving.
    • Allows registering custom compressors via RegisterCompressor.
    // Create archive file
    w, err := os.Create("archive.zip")
    if err != nil {
      panic(err)
    }
    defer w.Close()
    
    // Create new Archiver
    a, err := fastzip.NewArchiver(w, "~/fastzip-archiving")
    if err != nil {
      panic(err)
    }
    defer a.Close()
    
    // Register a non-default level compressor if required
    // a.RegisterCompressor(zip.Deflate, fastzip.FlateCompressor(1))
    
    // Walk directory, adding the files we want to add
    files := make(map[string]os.FileInfo)
    if err = filepath.Walk("~/fastzip-archiving", func(pathname string, info os.FileInfo, err error) error {
      files[pathname] = info
      return nil
    }); err != nil {
      panic(err)
    }
    
    // Archive
    if err = a.Archive(context.Background(), files); err != nil {
      panic(err)
    }
  3. Extract a Zip archive with Extractor

    main

    Use fastzip.NewExtractor to create an extractor instance. You must provide the path to the zip archive and a target directory for extraction. For security, extraction is restricted to the specified target directory.

    Key features:

    • Preserves permissions, ownership (uid, gid on Linux/Unix), and modification times.
    • Uses github.com/klauspost/compress/flate by default for high performance.
    • Supports concurrent file extraction.
    // Create new extractor
    e, err := fastzip.NewExtractor("archive.zip", "~/fastzip-extraction")
    if err != nil {
      panic(err)
    }
    defer e.Close()
    
    // Extract archive files
    if err = e.Extract(context.Background()); err != nil {
      panic(err)
    }
  4. Use ExtractorOption to configure an Extractor

    main

    The ExtractorOption type is a functional option used to configure an extractor during its creation. You can pass multiple options to the extractor constructor to customize its behavior, such as concurrency limits and error handling policies.

    type ExtractorOption func(*extractorOptions) error
    
    // Usage pattern:
    extractor, err := fastzip.NewExtractor(
        fastzip.WithExtractorConcurrency(8),
        fastzip.WithExtractorChownErrorHandler(myHandler),
    )
  5. Register custom compressors with RegisterCompressor

    main

    The Archiver comes with built-in support for zip.Deflate and zstd.ZipMethodWinZip. You can extend the archiver by registering custom compressors for specific method IDs using RegisterCompressor.

    RegisterCompressor(method uint16, comp zip.Compressor)

    • method: The ZIP compression method ID.
    • comp: A zip.Compressor implementation for that method.
    // Register a custom compressor
    archiver.RegisterCompressor(customMethodID, myCustomCompressor)
  6. Handle ownership errors with WithExtractorChownErrorHandler

    main

    Use WithExtractorChownErrorHandler(fn func(string, error) error) to define how the extractor should behave when errors occur while attempting to preserve file ownership (chown).

    • To ignore ownership errors and continue extraction, return nil from the handler function.
    • To abort the extraction process upon an ownership error, return an error from the handler function.
    // Example: Ignore chown errors to ensure extraction completes
    extractor, err := fastzip.NewExtractor(
        fastzip.WithExtractorChownErrorHandler(func(name string, err error) error {
            // Log the error but return nil to continue extraction
            fmt.Printf("Warning: failed to chown %s: %v\n", name, err)
            return nil
        }),
    )
  7. Initialize an Extractor with NewExtractorFromReader

    main

    Use NewExtractorFromReader when the zip archive is already in memory or provided via an io.ReaderAt. You must provide the total size of the archive.

    Note: Unlike NewExtractor, calling Close() on an extractor created this way is unnecessary because it does not own the underlying reader.

    // r is an io.ReaderAt, size is the archive size in bytes
    extractor, err := fastzip.NewExtractorFromReader(r, size, "/path/to/destination")
    if err != nil {
    	log.Fatal(err)
    }
    
    err = extractor.Extract(ctx)
  8. Initialize an Extractor with NewExtractor

    main

    Use NewExtractor to open a zip file from a filename and extract its contents to a specific chroot directory. The chroot directory acts as a security boundary; files cannot be extracted outside of it. You can provide optional ExtractorOption functions to configure behavior.

    Note: You must call Close() on the returned *Extractor to release the underlying zip reader resources.

    extractor, err := fastzip.NewExtractor("archive.zip", "/path/to/destination", fastzip.WithConcurrency(8))
    if err != nil {
    	log.Fatal(err)
    }
    defer extractor.Close()
    
    err = extractor.Extract(ctx)
  9. Register custom decompressors

    main

    By default, fastzip supports Store and Deflate methods, and includes built-in support for zstd (via zstd.ZipMethodWinZip). You can use RegisterDecompressor to add support for custom compression methods by mapping a method ID to a zip.Decompressor.

    // method is the uint16 method ID from the zip header
    extractor.RegisterDecompressor(customMethodID, myCustomDecompressor)
  10. Get a pooled StdFlate decompressor

    main

    Use StdFlateDecompressor() to obtain a factory function that returns a pooled decompressor using the standard library compress/flate implementation.

    The returned function has the signature func(r io.Reader) io.ReadCloser. Calling it retrieves a decompressor from a pool and resets it to the provided reader.

    Important: You must call .Close() on the returned io.ReadCloser to return the decompressor to the pool for reuse.

    // Using the standard library implementation
    decompressorFactory := fastzip.StdFlateDecompressor()
    reader, err := decompressorFactory(inputReader)
    if err != nil {
    	return err
    }
    defer reader.Close()
  11. Monitor progress with Written

    main

    The Written method allows you to monitor the progress of an ongoing archiving operation. It returns the total number of bytes and the number of entries written to the archive so far. This method is safe to call concurrently while Archive is running.

    Written() (bytes, entries int64)

    ```go
    // In a separate goroutine or loop
    gos.Go(func() {
    	for {
    		bytes, entries := archiver.Written()
    		fmt.Printf("Progress: %d bytes, %d entries\n", bytes, entries)
    		time.Sleep(time.Second)
    	}
    })
  12. Create a new Archiver with NewArchiver

    main

    Use NewArchiver to initialize an opinionated Zip archiver. The archiver requires an io.Writer for the output and a chroot directory. Only files, symlinks, and directories that are children of the chroot directory will be archived. Access permissions, Unix ownership, and modification times are preserved.

    NewArchiver accepts optional ArchiverOption functions to configure behavior such as compression method, concurrency, and buffer sizes.

    // Example initialization
    archiver, err := fastzip.NewArchiver(outputFile, "/path/to/chroot", fastzip.WithConcurrency(4))
    if err != nil {
    	log.Fatal(err)
    }
    defer archiver.Close()