SnapRAID Documentation

SnapRAID is a backup program for disk arrays designed for home media centers with large, infrequently changing files. It provides parity-based data recovery and integrity protection with the following characteristics:

• Data Recovery: Can recover data in the event of up to six disk failures (depending on parity configuration).
• Non-Destructive: Uses existing disks without requiring reformatting; disks can be accessed normally outside of SnapRAID.
• Data Integrity: All data is hashed to prevent silent corruption.
• Fault Isolation: If failures exceed the parity count, data loss is confined to the affected disks; other disks remain accessible.
• Flexibility: Supports disks of different sizes and allows adding new disks at any time.
• Low Power/Noise: Accessing a file only requires a single disk to spin up.
• No Lock-in: You can stop using SnapRAID at any time without moving or reformatting your data.

By enabling the snapshot option in the configuration, SnapRAID uses filesystem snapshots to ensure atomic and consistent operations during sync, scrub, check, and fix.

Benefits:

• Consistency: Prevents parity mismatches caused by concurrent file modifications on the live filesystem.
• Recovery: If a file is deleted or updated on the live filesystem during a process, it remains preserved in the snapshot, allowing SnapRAID to use it for data reconstruction.

Requirements & Support:

• Requires administrative privileges (e.g., sudo).
• Supported filesystems: Btrfs, Bcachefs, and ZFS on Linux; NTFS on Windows.
• Snapshots are only created for data disks; parity disks always use the live filesystem.

Lifecycle:

• stable snapshot: Represents the state of the last successful sync. Used by scrub, check, and fix.
• pending snapshot: A temporary image created at the start of a sync. It becomes the new stable snapshot upon successful completion.

SnapRAID uses glob patterns to filter files and directories.

Pattern Syntax

• ?: Matches any single character (excluding directory separators).
• *: Matches any string within a single directory level (does not cross directory boundaries).
• **: Matches any string across multiple directory levels. Use /**/ to match any depth (e.g., src/**/main.js).
• [ ]: Character classes (e.g., [a-z]) or negated classes (e.g., [!a-z]).

Pattern Types

• FILE: Matches a file name (not a directory).
• DIR/: Matches a directory and all its contents.
• /PATH/FILE: Matches an exact file path.
• /PATH/DIR/: Matches an exact directory path and all its contents.
• Note: Absolute paths starting with / are relative to the array root, not the system root.

Configuration Strategies

1. Exclude-only: List everything you want to remove using exclude.
2. Include-only: List only what you want to keep using include.
3. Mixed: Use both, but order matters. Earlier rules take precedence. Typically, list exclude rules first, then include rules.

Local Overrides

You can place a .snapraidignore file in any directory to define local exclusion rules. These are applied after global configuration rules, meaning they have higher priority and can exclude files even if they were included globally. .snapraidignore only supports exclusion rules.

SnapRAID uses snapshots to ensure data consistency and prevent parity mismatches caused by simultaneous file modifications. The behavior of commands depends on whether they target specific disks or the entire array:

• sync and scrub: These operate exclusively on snapshots. sync uses a new pending snapshot to calculate parity, while scrub uses the stable snapshot (from the last successful sync) as a reference.
• check and fix:
  • If a specific disk is targeted using -d or --filter-disk, SnapRAID uses the active filesystem for that disk. This allows fix to restore data to an active replacement disk and check to simulate a fix under real conditions.
  • For all other (reference) data disks, SnapRAID uses their snapshots to ensure a stable reference even if files are being modified elsewhere.
  • If no disk is specified with -d, check and fix use the active filesystems for the entire array.
• All other commands: Operate exclusively on the active filesystem.

SnapRAID uses globbing patterns to include or exclude files and directories. Patterns are processed in the order they appear in the configuration.

Pattern Logic:

• If a pattern matches and is an exclude, the file is excluded.
• If a pattern matches and is an include, the file is included.
• If no patterns match, the file is excluded if the last specified pattern was an include, or included if the last was an exclude.

Globbing Syntax:

• ?: Matches any single character (except directory separators).
• *: Matches any sequence of characters within a single directory level (does not cross directory boundaries).
• **: Matches any sequence of characters, including directory separators.
• /**/: A special form of ** that matches zero or more directory levels, allowing you to match files at any depth in a tree.

exclude /lost+found/
exclude /tmp/
include /home/user/**/*.jpg

SnapRAID uses globbing characters to include or exclude files and directories:

• ?: Matches any single character (excluding directory separators).
• *: Matches any sequence of characters within a single directory level (does not cross directory boundaries).
• **: Matches any sequence of characters, including directory separators. Use /**/ to match across any number of directory levels (e.g., src/**/main.js matches src/main.js, src/ui/main.js, etc.).
• []: Character classes that match a single character from a specified set or range. Supports negation with !.

Pattern Types:

• FILE: Matches all files named FILE (not directories).
• DIR/: Matches all directories named DIR and everything inside them.
• /PATH/FILE: Matches the exact file path (not directories).
• /PATH/DIR/: Matches the exact directory path and everything inside it.

Note on Absolute Paths: Paths starting with / are applied to the array root directory, not the local filesystem root.

Platform Specifics:

• Windows: You can use backslashes \ instead of forward slashes /. Windows system directories, junctions, and mount points are treated as files.
• Escaping: To use literal *, ?, [, or ] in filenames, escape them with \ on Unix or ^ on Windows. If the pattern is on the command line, you must double the escape character to prevent the shell from interpreting it.