zip.js

repository·master·Indexed 26 days ago

https://github.com/gildas-lormeau/zip.js

A high-performance JavaScript library for compressing and decompressing ZIP files in the browser, Deno, and Node.js. It supports multi-core compression, native compression streams, Zip64 for archives larger than 4GB, split zip files, data encryption, and Deflate64 decompression. The library is optimized for large datasets and modern web environments, providing streaming capabilities via ZipReader and ZipWriter.

Tokens
45.5K
Snippets
22
Records
348
Agent score
86%

What's inside zip.js

  1. Overview of zip.js capabilities

    master

    zip.js is a JavaScript library (BSD-3-Clause license) for compressing and decompressing zip files. It is designed for high performance and large datasets, supporting:

    • Multi-core compression
    • Native compression using compression streams
    • Archives larger than 4GB via Zip64
    • Split zip files
    • Data encryption
    • Incremental writing
    • Deflate64 decompression
  2. Explore @zip.js/zip.js API surface

    master

    The @zip.js/zip.js library provides a comprehensive set of tools for reading and writing ZIP files in JavaScript. The API is organized into several categories:

    Core Classes

    • Readers: ZipReader, ZipReaderStream, BlobReader, HttpReader, Uint8ArrayReader, and others for various data sources.
    • Writers: ZipWriter, ZipWriterStream, BlobWriter, Uint8ArrayWriter, and others.
    • Entries: ZipEntry, ZipFileEntry, and ZipDirectoryEntry for interacting with files and folders within an archive.

    Configuration and Interfaces

    • Options: Interfaces like ZipReaderOptions, ZipWriterConstructorOptions, and GetEntriesOptions define how to initialize and control archive operations.
    • Data Handling: Interfaces for managing streams, file system handles, and metadata.

    Error Handling

    • The library uses specific error constants (e.g., ERR_BAD_FORMAT, ERR_INVALID_PASSWORD, ERR_UNSUPPORTED_COMPRESSION) to identify issues during ZIP processing.
  3. Choose the right compression library for your workload

    master

    Based on benchmarked performance, select a library according to your specific use case:

    • Large files or multiple entries: Use @zip.js/zip.js. It is the fastest for compressing large or multiple entries and is the only library that parallelizes compression across CPU cores without requiring Web Workers (on Node and Bun) by leveraging the native CompressionStream.
    • Thousands of tiny files: Use fflate. It is the champion for throughput and low memory footprint when dealing with a high volume of small buffers/files.
    • Large file streaming: Use @zip.js/zip.js, fflate, or archiver. Avoid jszip for files that do not fit comfortably in memory, as it buffers the entire file.
    • Single-file streaming (speed vs ratio): @zip.js/zip.js is highly competitive against native tools like 7-Zip, often providing faster and tighter compression than 7-Zip's fast modes.
  4. Understand zip.js throughput and runtime dependencies

    master

    For bulk data, zip.js acts as a thin wrapper around the platform's CompressionStream. Consequently, zip.js throughput is primarily determined by the zlib implementation provided by your runtime:

    • Deno & Bun: Use zlib-ng, offering the highest throughput (e.g., ~4.8–5.1s for a 256 MB file).
    • Node.js: Uses a Chromium zlib fork (e.g., ~8.9s for a 256 MB file).
    • WASM Backend: If you disable CompressionStream (using useCompressionStream: false), zip.js uses a classic zlib compiled to WebAssembly. This is the 'portability floor' and is slower than native runtimes (e.g., ~16s for a 256 MB file) but remains competitive with native tools like 7-Zip.

    Key Takeaway: The same zip.js code typically runs ~1.75× faster on Deno/Bun than on Node due to the underlying zlib performance.

  5. Compare zip.js performance against other libraries

    master

    When choosing a ZIP library, consider these trade-offs based on your use case:

    • Use zip.js if you need the fastest compression for large or multiple entries (using parallelism without Web Workers), the fastest large-stream decompression, low-memory streaming of huge files, or advanced ZIP features like AES/ZipCrypto encryption, Zip64, and split/multi-volume archives.
    • Use fflate if you need to deflate thousands of tiny buffers in a single call and require the smallest memory footprint and highest raw synchronous throughput.
    • Use archiver if you need a solid streaming compressor on Node.js (note: it cannot read archives).
    • Use jszip for convenience, but be aware it is slower and buffers whole files in memory.
  6. Reproduce benchmarks locally

    master

    To run the benchmark suite on your own machine, follow these steps in the benchmarks/ directory:

    1. Install dependencies: npm install (installs jszip, fflate, and archiver).
    2. Generate datasets: npm run corpus.
    3. Run head-to-head tables (compress/decompress/disk streaming): node bench.js.
    4. Run parallelism & codec-backend matrix: node bench-backends.js.
    5. Compare against 7-Zip (requires 7-Zip CLI installed): deno run -A bench-7z.js or bun bench-7z.js.

    Environment Variables & Flags:

    • ZIPJS_BACKEND=wasm: Forces the use of the WebAssembly codec instead of the native CompressionStream during 7-Zip comparisons.
    • SKIP_HUGE=1: Skips the 256 MB combination test.
    • RUNS=<n>: Sets the number of repetitions (default is 3).
    cd benchmarks
    npm install            # jszip, fflate, archiver (zip.js is used from the repo)
    npm run corpus         # generate the deterministic datasets under .corpus/
    node bench.js          # the head-to-head tables (compress / decompress / disk streaming)
    node bench-backends.js # the parallelism & codec-backend matrix
    deno run -A bench-7z.js # zip.js vs the 7zz CLI (also: bun bench-7z.js)
  7. Optimize @zip.js/zip.js for maximum throughput

    master

    To ensure you are using the fastest possible configuration in @zip.js/zip.js:

    1. Use the default compression level: Requesting a custom level forces a fallback to the WASM zlib codec, which prevents the native CompressionStream from parallelizing via concurrent add() calls.
    2. Use concurrent add() calls: Instead of adding entries sequentially, call add() multiple times concurrently to leverage the platform's threadpool.
    3. Configure Web Workers for specific runtimes: If running in Deno or Safari/WebKit, ensure you pass useWebWorkers: true to enable true parallelism.
  8. Enable parallelism in @zip.js/zip.js

    master

    To achieve maximum compression speed via parallelism, you can use concurrent add() calls. How you configure this depends on your runtime:

    • Node.js and Bun: Simply issuing concurrent add() calls is sufficient. These runtimes back CompressionStream with a threadpool, allowing independent entries to compress on multiple cores without Web Workers.
    • Deno and Safari/WebKit: You must set useWebWorkers: true in your configuration. These runtimes run CompressionStream on the main/isolate thread, so concurrent add() calls alone will not provide a speedup.

    Important Caveat: Parallelism via concurrent add() only works with the native CompressionStream backend. If you request a non-default compression level (e.g., { level: 5 }), zip.js falls back to the WASM zlib codec, which runs synchronously on the main thread and does not benefit from concurrent add() calls.

  9. Configure WebAssembly and Web Worker URIs

    master

    To avoid Content Security Policy (CSP) issues, you can import the WASM module and the worker script as URLs (using tools like Vite's ?url suffix) and provide them to the configure() function via wasmURI and workerURI.

    import wasmURI from "@zip.js/zip.js/dist/zip-module.wasm?url";
    import workerURI from "@zip.js/zip.js/dist/zip-web-worker.js?url";
    
    configure({
      wasmURI,
      workerURI
    });
  10. Configure Zip64 and USDZ compatibility

    master

    Handle large files and specific formats:

    • zip64: Set to true to use Zip64. Automatically set to true if compressed data exceeds 4GB or has an unknown size. Defaults to false.
    • supportZip64SplitFile: Set to false to never write disk numbers in zip64 data. Defaults to true.
    • usdz: Set to true to produce zip files compatible with the USDZ specification. Defaults to false.
  11. Configure encryption and password settings in ZipWriterConstructorOptions

    master

    You can encrypt zip entries using the following options:

    • password: A string used to encrypt the content.
    • rawPassword: A Uint8Array (or ArrayBufferLike) used for raw password encryption.
    • encrypted: Set to true to write encrypted data when passThrough is also true.
    • encryptionStrength: The AES encryption strength. Options are 1 (128-bit), 2 (192-bit), or 3 (256-bit). Defaults to 3.
    • zipCrypto: A boolean to use the ZipCrypto algorithm. Note: This is not recommended as it is easily broken. Setting this to true automatically sets dataDescriptor to true.
  12. Configure timestamps and Unicode filenames

    master

    Manage file metadata and encoding:

    • creationDate, lastAccessDate, lastModDate: Date objects for file timestamps. These are ignored if extendedTimestamp is false.
    • extendedTimestamp: Set to true to store extended timestamp extra fields. If false, dates are limited to a max accuracy of 2 seconds and a max date of November 31, 2107.
    • useUnicodeFileNames: Set to true to mark filenames as UTF-8 (setting general purpose bit 11). Defaults to true.