unionfs

repository·master·Indexed 42 days ago

https://github.com/streamich/unionfs

A Node.js library (v4.6.0) that allows developers to create a single unified file system interface by layering multiple fs-compatible objects. It supports combining real disk storage with in-memory file systems like memfs or memory-fs, providing a priority-based search order where the most recently added filesystem via the .use() method takes precedence.

Tokens
2.9K
Snippets
12
Records
15
Agent score
52%

What's inside unionfs

  1. Create a union of multiple file systems

    master

    You can use the exported ufs instance to layer multiple file systems. The order of .use() calls determines the priority: file systems added later will shadow or overlay those added earlier if they contain the same paths.

    unionfs works with any object that implements the standard fs API, such as memfs (in-memory), linkfs (path redirection), or memory-fs.

    import { ufs } from 'unionfs';
    import { fs as fs1 } from 'memfs';
    import * as fs2 from 'fs';
    
    // Layer fs1 on top of fs2
    ufs.use(fs1).use(fs2);
    
    // Operations on ufs will now search through the layered file systems
    ufs.readFileSync(/* ... */);
  2. How the Union filesystem stack works

    master

    A Union object represents a stack of filesystems. When performing operations, it follows these principles:

    1. Priority: The search order is determined by the order in which filesystems are added. The most recently added filesystem (via .use()) is treated as the first filesystem when searching for a file.
    2. Method Proxying: The Union class dynamically implements most standard Node.js fs methods (both synchronous and asynchronous) by delegating them to the underlying filesystems in the stack.
    3. Error Handling: For most methods, if one filesystem fails, the Union instance will catch the error and attempt the operation on the next filesystem in the stack. If all filesystems fail, the last error encountered is thrown. Errors may include a prev property to trace the error chain.
    4. Read/Write Streams: createReadStream and createWriteStream search through the stack to find the first available filesystem that can satisfy the request.
  3. Use multiple in-memory volumes with unionfs

    master

    This example demonstrates how to combine multiple memfs volumes and a memory-fs instance into a single unified file system interface.

    import * as fs from 'fs';
    import { Volume } from 'memfs';
    import * as MemoryFileSystem from 'memory-fs';
    import { ufs } from 'unionfs';
    
    const vol1 = Volume.fromJSON({ '/memfs-1': '1' });
    const vol2 = Volume.fromJSON({ '/memfs-2': '2' });
    
    const memoryFs = new MemoryFileSystem();
    memoryFs.writeFileSync('/memory-fs', '3');
    
    // Layering the file systems
    ufs.use(fs).use(vol1).use(vol2).use(memoryFs);
    
    console.log(ufs.readFileSync('/memfs-1', 'utf8')); // 1
    console.log(ufs.readFileSync('/memfs-2', 'utf8')); // 2
    console.log(ufs.readFileSync('/memory-fs', 'utf8')); // 3
  4. Create a manual Union instance

    master

    Instead of using the singleton ufs instance, you can instantiate your own Union class to manage isolated sets of file systems.

    import { Union } from 'unionfs';
    
    var ufs1 = new Union();
    ufs1.use(fs).use(vol);
    
    var ufs2 = new Union();
    ufs2.use(fs).use(/*...*/);
  5. Use `watch()` to monitor multiple filesystems

    master

    The watch() method allows you to monitor file changes across all attached filesystems. It returns a proxy object that, when a method is called on it, executes that method on all active watchers collected from the underlying filesystems.

    const watchers = ufs.watch('path/to/file');
    // Calling a method on the returned proxy calls it on all underlying watchers
    watchers.close();
  6. Define a compatible File System interface with IFS

    master

    To use a custom file system with unionfs, your implementation must satisfy the IFS interface. This interface is a subset of the Node.js fs module, including both synchronous and asynchronous methods, as well as the promises API. It also requires support for specific constants and stream types.

    An IFS implementation must include:

    • Core fs methods (e.g., stat, readFile, mkdir, readdir, etc.)
    • The promises object for Promise-based file operations.
    • Constants like F_OK, R_OK, W_OK, and X_OK.
    • Stats and Dirent types/classes.
    • ReadStream and WriteStream compatible constructors or types.
    import { IFS } from 'unionfs/src/fs';
    
    // Example of a minimal structure for an IFS implementation
    const myCustomFS: IFS = {
      // ... implement required fs methods
      promises: { /* ... */ },
      F_OK: undefined,
      R_OK: undefined,
      W_OK: undefined,
      X_OK: undefined,
      constants: undefined,
      Stats: undefined,
      Dirent: undefined,
      ReadStream: Readable,
      WriteStream: Writable,
    };
  7. Add filesystems to a Union instance using `use()`

    master

    The use() method adds a filesystem implementing the IFS interface to the Union stack. Filesystems added later are searched first (they are pushed to the end of the internal array, but the search logic iterates from the end of the array to the beginning, effectively treating the most recently added filesystem as the highest priority).

    const ufs = new Union();
    ufs.use(fs1).use(fs2);
  8. Reference: Union filesystem methods

    master

    The Union class implements a wide range of standard Node.js fs methods by delegating to the underlying IFS implementations. These include:

    • Synchronous Methods: existsSync, readdirSync, and others from fs-monkey/lib/util/lists.
    • Asynchronous (Callback) Methods: Standard fs callback-style methods.
    • Promise-based Methods: Methods available on fs.promises (e.g., access, copyFile, mkdir, readFile, writeFile, etc.).
    • Special Methods: watch, watchFile, unwatchFile, createReadStream, and createWriteStream which have custom union logic.
  9. Reference: Supported FS methods for IFS

    master

    The following methods are required for an object to be considered a valid IFS implementation. This includes a mix of synchronous and asynchronous versions of standard Node.js fs operations.

    type FSMethods =
      | 'renameSync'
      | 'ftruncateSync'
      | 'truncateSync'
      | 'chownSync'
      | 'fchownSync'
      | 'lchownSync'
      | 'chmodSync'
      | 'fchmodSync'
      | 'lchmodSync'
      | 'statSync'
      | 'lstatSync'
      | 'fstatSync'
      | 'linkSync'
      | 'symlinkSync'
      | 'readlinkSync'
      | 'realpathSync'
      | 'unlinkSync'
      | 'rmdirSync'
      | 'mkdirSync'
      | 'readdirSync'
      | 'closeSync'
      | 'openSync'
      | 'utimesSync'
      | 'futimesSync'
      | 'fsyncSync'
      | 'writeSync'
      | 'readSync'
      | 'readFileSync'
      | 'writeFileSync'
      | 'appendFileSync'
      | 'existsSync'
      | 'accessSync'
      | 'createReadStream'
      | 'createWriteStream'
      | 'watchFile'
      | 'unwatchFile'
      | 'watch'
      | 'rename'
      | 'ftruncate'
      | 'truncate'
      | 'chown'
      | 'fchown'
      | 'lchown'
      | 'chmod'
      | 'fchmod'
      | 'lchmod'
      | 'stat'
      | 'lstat'
      | 'fstat'
      | 'link'
      | 'symlink'
      | 'readlink'
      | 'realpath'
      | 'unlink'
      | 'rmdir'
      | 'mkdir'
      | 'readdir'
      | 'close'
      | 'open'
      | 'utimes'
      | 'futimes'
      | 'fsync'
      | 'write'
      | 'read'
      | 'readFile'
      | 'writeFile'
      | 'appendFile'
      | 'exists'
      | 'access';
  10. Instantiate a new `Union` class

    master

    If you need multiple independent union file system instances, you can instantiate the Union class directly. The resulting instance implements IUnionFs.

    import { Union } from 'unionfs';
    
    const myUnion = new Union();
    myUnion.use(fs1);
    myUnion.use(fs2);
  11. Handle errors in Union operations with `IUnionFsError`

    master

    When an operation fails across the entire filesystem stack, the Union instance throws an error that implements the IUnionFsError interface. This error includes a prev property, allowing you to traverse the chain of errors encountered in each filesystem.

    export interface IUnionFsError extends Error {
      prev?: IUnionFsError | null;
    }