DOCSEARCHbyAlgolia
Submit Docs

mergerfs Documentation

repository·master·Indexed 26 days ago

https://github.com/trapexit/mergerfs

A FUSE-based union filesystem that logically combines multiple storage paths (branches) into a single mount point. Designed for managing large collections of files across commodity drives without RAID, it supports dynamic management, configurable placement policies, and File IO passthrough. It provides features such as hard link copy-on-write (CoW), POSIX ACLs, and extended attributes, while explicitly not providing data redundancy, parity, or file splitting.

Tokens
32.2K
Snippets
58
Records
213
Agent score
90%

What's inside mergerfs

  1. Overview of mergerfs union filesystem

    master

    mergerfs is a union filesystem that logically combines multiple filesystems and/or directories into a single unified mount point. It acts as a smart pooling layer (JBOFS: Just a Bunch of FileSystems) that allows you to manage files across multiple independent storage devices (HDDs, SSDs, network shares, etc.) without the complexity or fragility of RAID.

    Key Characteristics:

    • Non-destructive: You can add or remove filesystems/paths at any time (even at runtime) without restructuring the pool or impacting existing data.
    • Fault Tolerance: Unlike RAID, a single device failure only results in the loss of files on that specific filesystem; the rest of the pool remains accessible.
    • Direct Access: You maintain direct access to files on the individual underlying filesystems.
    • Flexible Placement: Uses configurable policies to determine where new files are created.
    • Performance: Supports File IO passthrough for near-native performance where supported.

  2. Overview of mergerfs

    master
    mergerfs is a FUSE-based union filesystem designed to simplify the storage and management of files across multiple commodity storage devices. It logically combines multiple filesystem paths (branches) into a single mount point. Unlike RAID, it does not split files across branches or provide parity; instead, it acts as a proxy that selects which underlying path to use for specific operations based on configurable policies.

  3. Understand mergerfs resource management and limits

    master

    mergerfs performs several automatic management actions to ensure stability and performance:

    • OOM Killer Protection: To reduce the risk of the Linux kernel's OOM Killer targeting the process, mergerfs sets its oom_score_adj value to -990.
    • Resource Limits: mergerfs automatically increases its available file descriptor and file size limits (setrlimit).
    • CPU Priority: By default, mergerfs sets its scheduling priority (nice value) to -10, which is higher than the default system priority of 0.
    • Readahead: The readahead values for both mergerfs itself and the managed filesystems can be modified.

  4. Understand mergerfs terminology

    master

    To effectively use and configure mergerfs, it is important to distinguish between its core concepts:

    • Filesystem: The primary unit mergerfs interacts with. mergerfs operates on filesystems, not physical disks.
    • Branch: A base path used within a mergerfs pool. A single filesystem can have multiple branches.
    • Pool: The resulting mergerfs mount, representing the union of all branches. You can create multiple pools even using the same branches.
    • Path: A location within a filesystem. mergerfs can work with any path, not just the root.
    • Relative Path: The path within the pool relative to the branch and mount (e.g., if the mount is /mnt/mergerfs/foo/bar, the relative path is foo/bar).
    • Policy: The algorithm used to select specific files or directories when performing a filesystem function.
    • Function: A filesystem call (e.g., open, unlink, create, getattr, rmdir) made by software to the filesystem.
    • Category: A grouping of functions based on behavior (action, create, search).
    • Path Preservation: A feature of certain policies that checks the path where a file would be created.
    • Out-of-band: Interacting with the underlying filesystems directly, bypassing the mergerfs mount (and any network protocols like NFS or Samba).

  5. Supported Operating Systems for mergerfs

    master

    mergerfs is primarily designed for Linux.

    • FreeBSD: Casually supported but not well tested. Note that certain Linux functions and FUSE features are not supported on FreeBSD, which may limit advanced features.
    • MacOS/Windows: Not officially supported. For Windows, Drive Pool is suggested as an alternative. mergerfs may potentially be used with WSL on Windows.

  6. Understand mergerfs resource usage

    master

    mergerfs consumes resources in the following ways:

    Threads

    You can configure the number of threads used for:

    • Reading from the kernel.
    • Processing messages from the kernel.
    • readdir concurrency.

    Memory

    Memory consumption is influenced by:

    • Pre-reader thread and inflight processing: Uses 1MB+ depending on the fuse-msg-size configuration.
    • Directory reading: Temporary buffers are allocated when reading directories.
    • FUSE nodes: Memory is used for FUSE nodes.
    • Node retention: If never-forget-nodes=true is set, nodes are retained until they are unlinked or removed.

  7. Enable near-native IO performance with preload.so

    master

    The preload.so library is an experimental tool designed to simulate IO passthrough in FUSE. It intercepts file operations (open, creat, fopen), identifies the underlying branch, and reopens the file directly on the native filesystem to achieve near-native read/write performance.

    Important Limitations:

    • Feature Loss: Certain mergerfs features that interrupt the IO workflow, such as moveonenospc, will not work.
    • Compatibility: Only works with dynamically linked software using a compatible libc version. It will not work with statically compiled software (e.g., many Go or Rust applications).
    • Fallback: The library is designed to fallback to the standard mergerfs opened file if an error occurs, ensuring it does not interfere with non-mergerfs filesystems.

  8. How mergerfs works

    master

    mergerfs operates as a proxy for filesystem paths rather than block devices. It merges multiple paths logically:

    1. Directory Listing: When a directory is requested, mergerfs combines the file lists from all branches, deduplicates entries, and returns the unified list.
    2. File Creation: When a file or directory is created, mergerfs runs a configured policy to select which branch (underlying filesystem) will host the new data.
    3. Attribute/Removal Operations: For functions that change attributes or remove files, the behavior may be applied to all instances found across the branches.

    Behavior is controlled via config/options/settings and specifically through policies.

  9. Understand mergerfs decision-making context

    master

    When making filesystem decisions (such as file placement or permission handling), mergerfs utilizes the following information from incoming requests:

    • The tid or pid of the calling thread/process
    • The uid and gid of the calling thread/process
    • The umask of the calling thread/process
    • The arguments of the filesystem call
    • The mergerfs configuration
    • Information queryable from the operating system or underlying filesystems

  10. Assess mergerfs reliability for production use

    master

    mergerfs is considered production-ready and has been used for over a decade in environments with constant 24/7 loads. It is included in several NAS-focused operating systems as a filesystem pooling solution.

    Note on Stability: Most serious issues such as crashes or data corruption are typically attributed to Linux kernel bugs rather than mergerfs itself. Users should ensure they are using stable releases and are aware of known FUSE/kernel interactions.

  11. Interact with the mergerfs Runtime Interface

    master

    mergerfs provides a runtime interface via the POSIX extended attributes (xattr) API. This allows you to query filesystem information, get/set configuration, and trigger commands while the filesystem is running.

    There are two primary targets for xattr calls:

    1. The <mountpoint>/.mergerfs pseudo file: Used for getting/setting configuration and issuing commands.
    2. Files/directories on the filesystem: Used to query mergerfs-specific metadata about those files.

    Warning: Changes made via the .mergerfs pseudo file are NOT persisted. To make changes permanent, you must update your mount configuration (e.g., /etc/fstab or systemd).

  12. Key features of mergerfs

    master

    mergerfs provides several capabilities for managing pooled storage:

    • Path Merging: Combine multiple paths (from the same or different filesystems) into one mount point.
    • Dynamic Management: Add or remove filesystems/paths without impacting existing data.
    • Resilience: The union mount remains unaffected by the failure of individual underlying filesystems.
    • Configurable Placement: Use policies to control file selection and where new files are created.
    • Performance: Supports File IO passthrough for near-native performance where supported.
    • Compatibility: Works with filesystems of any size and almost any type.
    • Advanced Attributes: Supports extended attributes (xattrs), file attributes (chattr), and POSIX ACLs.
    • Link Support: Supports hard links and hard link copy-on-write (CoW).