yazl

repository·master·Indexed 18 days ago

https://github.com/thejoshwolfe/yazl

A streaming ZIP creation library for Node.js designed to be memory-efficient and non-blocking. It provides the ZipFile class to create archives by adding files from the filesystem, buffers, or read streams, and supports ZIP64 and Info-ZIP universal timestamps.

Tokens
3.9K
Snippets
15
Records
20
Agent score
14%

What's inside yazl

  1. Overview of yazl

    master

    yazl is a Node.js library for creating ZIP files. It is designed with three core principles to ensure performance and stability in Node.js environments:

    1. Non-blocking: Uses asynchronous APIs to avoid blocking the JavaScript thread.
    2. Memory Efficient: Avoids buffering entire files in RAM; it processes data in a streaming fashion.
    3. Resource Conscious: Prefers opening input files one at a time to avoid hitting OS limits on simultaneous open file handles.
  2. How directory entries are handled

    master

    yazl handles directory structures in two ways:

    1. Implicit Directories: When you add a file using a metadataPath like "parent/file.txt", yazl does not automatically create a directory entry for "parent/". It assumes the unzip client will handle the creation of parent directories implied by the file paths.
    2. Explicit Empty Directories: If you need to include a directory that contains no files, you must explicitly call addEmptyDirectory().

    If you use metadataPath with backslashes (\), yazl will automatically replace them with forward slashes (/) to ensure standard pathing.

  3. How Info-ZIP universal timestamps work

    master

    Since version 3.3.0, yazl includes the Info-ZIP "universal timestamp" extended field (0x5455 or "UT") to encode mtime. This is the recommended modern encoding.

    Benefits:

    • Timezone is explicitly UTC (ideal for cloud/multi-timezone teams).
    • Can encode the Unix epoch (1970) accurately.
    • 1-second precision (vs. rounding to the nearest even second in DOS).

    Trade-offs:

    • Adds 9 bytes of metadata per entry.
    • If you need to revert to the older DOS-only behavior, set forceDosTimestamp: true in your file options.
  4. Understand ZIP64 support in yazl

    master

    yazl automatically switches to the ZIP64 format when an archive exceeds the limits of the original ZIP specification. This occurs if:

    • Files or the total archive size exceeds 2^32 - 2 bytes (~4GB).
    • The archive contains more than 2^16 - 2 (65,534) files.

    Note on Compatibility: While most modern zipfile readers support ZIP64, some tools like the Mac Archive Utility may not handle ZIP64 archives correctly. You can use the forceZip64Format option in the API to manually control this behavior.

  5. Create a ZIP file with yazl

    master

    To create a ZIP file, instantiate a new yazl.ZipFile, add your files or buffers using the provided methods, pipe the outputStream to a writable destination (like a file stream), and finally call .end() to finalize the archive.

    Note: When using addFile, you can only add files, not directories. To include a file at a specific path within the ZIP, provide the desired internal path as the second argument.

    var yazl = require("yazl");
    var fs = require("fs");
    
    var zipfile = new yazl.ZipFile();
    
    // Add a file from the local filesystem
    zipfile.addFile("file1.txt", "file1.txt");
    
    // Add a file with a specific internal path
    zipfile.addFile("path/to/file.txt", "path/in/zipfile.txt");
    
    // Add a Buffer
    zipfile.addBuffer(Buffer.from("hello"), "hello.txt");
    
    // Add a file using a lazy ReadStream (useful for streams like stdin)
    zipfile.addReadStreamLazy("stdin.txt", cb => cb(null, process.stdin));
    
    // Pipe the output to a file
    zipfile.outputStream.pipe(fs.createWriteStream("output.zip")).on("close", function() {
      console.log("done");
    });
    
    // Finalize the ZIP file
    zipfile.end();
  6. Use the ZipFile class to create zip files

    master

    The ZipFile class is the primary interface for creating ZIP archives. You instantiate it with new ZipFile() and then add files, buffers, or streams to it. The resulting archive is produced via the outputStream property, which is a readable stream. It is common practice to pipe this outputStream to a writable stream, such as one created by fs.createWriteStream().

    Important Note on Streams: Avoid using both .on('data') and .pipe() on the outputStream simultaneously, as this can cause incorrect behavior in certain Node.js versions.

    const yazl = require('yazl');
    const fs = require('fs');
    
    const zipfile = new yazl.ZipFile();
    zipfile.addFile('/path/to/real/file.txt', 'relative/path/in/zip.txt');
    zipfile.end();
    
    zipfile.outputStream.pipe(fs.createWriteStream('archive.zip'));
  7. Add content from a buffer with addBuffer()

    master

    Adds a file to the zip archive using a provided Buffer.

    Compatibility Note: If you need to create zip files that are compatible with the bug in 7-Zip 9.20 (which incorrectly handles General Purpose Bit 3), use addBuffer() for all entries in your archive.

    Size Limitation: To prevent issues with zlib inflation and Node.js buffer limits, yazl enforces that the provided buffer must be at most 0x3fffffff bytes long.

    Options:

    • All options available in addFile() (except size).
    const buf = Buffer.from('hello world');
    zipfile.addBuffer(buf, 'hello.txt', { compress: true });
  8. Add a file from the file system with addFile()

    master

    Adds a file located at realPath on your local file system into the zip archive at the specified metadataPath.

    Metadata Path Rules:

    • Must not be blank.
    • Backslashes (\) are automatically replaced with forward slashes (/).
    • Must not start with / or a drive letter pattern like /[A-Za-z]:\//.
    • Must not contain .. path segments.
    • Must not end with / (use addEmptyDirectory() for directories).
    • After UTF-8 encoding, the path must be at most 0xffff bytes.

    Options:

    • mtime: Override the file's modification time.
    • mode: Override the Unix permission bits and file type.
    • compress: Boolean. If true (default), uses deflate. If false, stores the file without compression.
    • compressionLevel: Integer (0-9) passed to zlib.
    • forceZip64Format: Boolean. If true, forces ZIP64 even if not strictly needed.
    • forceDosTimestamp: Boolean. If true, disables the modern Info-ZIP "universal timestamp" (UT) field, reverting to older behavior.
    • fileComment: A string or UTF-8 Buffer (max 0xffff bytes) for the entry's comment.
    zipfile.addFile(realPath, metadataPath, {
      mtime: new Date(),
      mode: 0o644,
      compress: true,
      compressionLevel: 6,
      fileComment: "This is a file comment"
    });
  9. Add content from a stream with addReadStreamLazy()

    master

    To avoid holding many system resources open for long periods, it is recommended to use addReadStreamLazy() instead of addReadStream(). This method takes a function that provides the read stream only when needed.

    Usage: Pass a function getReadStreamFunction(cb) where cb is a callback that receives (err, readStream). If an error is passed to the callback, it will be emitted from the ZipFile object.

    Options:

    • All options available in addFile().
    • size: The expected size in bytes. If provided, yazl will emit an error if the actual number of bytes read from the stream does not match this value.
    zipfile.addReadStreamLazy("path/in/archive.txt", function(cb) {
      var readStream = getTheReadStreamSomehow();
      cb(null, readStream);
    });
  10. Add an empty directory with addEmptyDirectory()

    master

    Adds an entry to the zip file that represents a directory. This is required if you want the archive to contain a directory that has no files inside it.

    Note: If the metadataPath does not end with a /, yazl will automatically append one.

    Options:

    • mtime: Modification time.
    • mode: Unix permissions (default 040775).
    • forceDosTimestamp: Boolean.
    zipfile.addEmptyDirectory('my-empty-folder/');
  11. Finalize the archive with end()

    master

    Signals that no more files will be added and triggers the closing of the outputStream.

    Options:

    • forceZip64Format: Boolean.
    • comment: A string or CP437 encoded Buffer for the entire ZIP file comment (max 0xffff bytes). For maximum compatibility, use only printable ASCII characters (0x20...0x7e).

    Calculating Total Size: If you need to know the final size of the ZIP file (e.g., to set a Content-Length header in a web server) before the stream is fully consumed, provide a calculatedTotalSizeCallback as the second argument.

    • If the callback returns -1, the size is unknown.
    • To ensure a known size, you must:
      1. Disable compression (compress: false or compressionLevel: 0) for all entries.
      2. Specify the size option in every addReadStream or addReadStreamLazy call.
    zipfile.end({ comment: "Archive complete" }, (totalSize) => {
      console.log(`The total zip size is ${totalSize} bytes`);
    });
  12. Reference: yazl Output Structure and Bit Flags

    master

    For developers implementing unzip tools or analyzing yazl's output, the following technical specifications apply:

    • Disk Numbers: All disk-related values are 0 (except the Total Number of Disks in the ZIP64 End of Central Directory Locator, which is 1).
    • Version Made By: Always 0x033f (UNIX, spec version 6.3).
    • Version Needed to Extract: Usually 20 (2.0) for UTF-8 support. When ZIP64 is used, some entries may be 45 (4.5).
    • General Purpose Bit Flag:
      • Bit 11 is always set (UTF-8 encoding for filenames/comments).
      • Bit 3 is set in the Local File Header to support streaming (Data Descriptors). yazl includes the optional signature in the Data Descriptor to maintain compatibility with Mac Archive Utility.
    • External File Attributes: Set to stats.mode << 16 (UNIX convention).
    • Internal File Attributes: Always 0 (files are treated as binary).