DOCSEARCHbyAlgolia
Submit Docs

ProxSave Documentation

repository·main·Indexed 19 days ago

https://github.com/tis24dev/proxsave

A tool for Proxmox PVE and PBS users to automate the backup and restoration of system files for disaster recovery and environment migrations. Features include scheduled automatic backups, multi-path storage support, encryption, intelligent retention, and integrated notifications via Telegram, email, webhooks, Gotify, and Prometheus. Includes a resident daemon mode with systemd integration and a streamed backup UI for real-time monitoring.

Tokens
110.4K
Snippets
224
Records
393
Agent score
67%

What's inside ProxSave

  1. Overview of ProxSave features

    main

    ProxSave is designed to simplify the recovery and migration of Proxmox Virtual Environment (PVE) and Proxmox Backup Server (PBS) environments. It allows you to save your entire environment configuration and restore it to a new installation with minimal manual changes.

    Key Features:

    • Automatic Backups: Scheduled saves of system files.
    • Multi-path Saves: Support for multiple backup destinations.
    • Intelligent Retention: Managed backup lifecycle.
    • Encryption: Secure backup storage.
    • Notifications: Integrated support for Telegram, email (via cloud relay or Proxmox Notifications), webhooks, Gotify, and Prometheus.

  2. When to use the Proxmox VE Cluster Recovery Guide

    main

    Use the advanced recovery procedures in this guide when facing:

    • Complete node failure requiring cluster database restoration.
    • Cluster corruption that prevents normal operation.
    • Hardware migration of a cluster node.
    • Disaster recovery from a backup following a catastrophic failure.

    Do NOT use this guide for:

    • Normal VM/CT backup and restore (use vzdump or Proxmox Backup Server).
    • Simple configuration changes (use the Web UI or /etc/pve).
    • Initial cluster creation (use pvecm create).

  3. What is included and excluded in a restore

    main

    Proxsave is designed for configuration restoration, not full system imaging.

    Included in restore:

    • System configurations (network, SSH, SSL, services).
    • Proxmox-specific configs (cluster, storage, datastores).
    • Custom scripts and cron jobs.
    • ZFS configurations and pool cache.
    • Backup jobs and scheduled tasks.

    NOT included in restore:

    • VM/CT disk images: Use Proxmox native tools for these.
    • Application data: Databases and app state under /var/lib/* are not restored.
    • System packages: Use apt or dpkg to reinstall software.
    • Direct pmxcfs writes: Proxsave uses a staged apply (API or controlled file apply) rather than direct tar extraction to /etc/pve.

  4. Overview of Proxsave Encryption

    main

    Proxsave uses the age encryption format (via filippo.io/age) to secure backup archives. It is designed to be a modern, secure replacement for GPG.

    Key Security Characteristics:

    • Streaming Encryption: The archive is encrypted as it is written. A plaintext archive file is never created on disk; instead, the data is streamed through a pipe from the compressor to the age writer.
    • Multiple Recipients: Supports both passphrase and key-based encryption. Any configured recipient can be used to decrypt a single backup.
    • Memory Safety: Sensitive data is zeroed immediately after use.
    • Standard Format: Fully compatible with standard AGE tools.

  5. Overview of Proxsave command-line structure and configuration

    main

    The proxsave binary (located at /opt/proxsave/build/proxsave) uses a standard command-line structure. You can combine flags and options to customize behavior.

    Command structure:

    proxsave [FLAGS] [OPTIONS]

    Configuration Precedence: When setting parameters, Proxsave resolves values in the following order (highest priority first):

    1. Command-line flags
    2. Environment variables
    3. Configuration file (default: configs/backup.env, resolved relative to the install directory, e.g., /opt/proxsave/configs/backup.env)
    4. Default values

  6. How PBS Datastore Mount Guards protect the root filesystem

    main

    When restoring Proxmox Backup Server (PBS) configurations, ProxSave uses Mount Guards to prevent data from being written to the host's root filesystem if a datastore's backing storage is not yet mounted. Without this, a restore might write into an empty mount-point directory on /, filling the root partition.

    How it works:

    • Detection: ProxSave checks if the target mount point is defined in fstab and if the real storage is currently mounted.
    • Protection: If the target is offline, ProxSave creates a read-only bind-mount using a guard directory with flags RO + nodev + nosuid + noexec. This ensures writes cannot land on the root FS.
    • Shadowing: When the real datastore mount comes online, it stacks on top of the guard (overlay). The datastore becomes usable, but the guard remains shadowed.
    • Fallback: If a bind-mount cannot be created, ProxSave logs a warning and proceeds without a guard. (Note: Older versions used chattr +i immutable flags, but this has been replaced by bind-mounts to avoid reboot-related issues).

    Cleanup: Use the --cleanup-guards flag to remove these protections.

  7. Understand the Proxsave Restore Workflow

    main

    The restore process is an orchestrated workflow that moves from preparation to execution through several distinct stages. When you trigger a restore (e.g., via the --restore flag), the system follows this lifecycle:

    1. Preparation: Decrypts the backup bundle, detects the target system type, validates compatibility, and analyzes the archive categories.
    2. Selection: The user selects a Restore Mode (Full, Storage, Base, or Custom) and specific Categories (e.g., SSH, Network, ZFS). A restore plan is then built.
    3. Cluster Mode Selection: If a cluster payload is detected, the user must choose between:
      • SAFE: Exports configuration to a directory and applies changes via API (VM/CT configs, storage, datacenter).
      • RECOVERY: Performs a full system restore.
    4. Confirmation: The user reviews the final plan and must explicitly type RESTORE to proceed.
    5. Safety & Service Management: The system creates safety backups of files to be overwritten and stops necessary services (PVE cluster services or PBS services) to ensure data integrity.
    6. Extraction: Files are extracted either to their original system paths (Normal categories) or to a timestamped directory (Export-only categories).
    7. Post-Restore: The system recreates storage directories, applies network/firewall/HA configurations, and checks ZFS pools if applicable.

  8. Choose between Cloud Layout: Bundle vs Raw

    main

    ProxSave supports two different ways to organize files in your cloud remote. This is controlled by the BUNDLE_ASSOCIATED_FILES environment variable.

    Bundle Mode (BUNDLE_ASSOCIATED_FILES=true - Default)

    Each backup is packed into a single <archive>.bundle.tar file.

    • Cloud Content: One file per backup.
    • Contents: The raw archive plus its .metadata and .sha256 sidecars.
    • Note: The .manifest.json is not inside the bundle.

    Raw Mode (BUNDLE_ASSOCIATED_FILES=false)

    Each component of the backup is uploaded as a separate file.

    • Cloud Content: Multiple files per backup: <archive>, <archive>.sha256, <archive>.manifest.json, <archive>.metadata, and <archive>.metadata.sha256.
    • Critical Requirement: You must not delete the <archive>.manifest.json file. It is the authoritative metadata required for discovery and verification during restore/decrypt cloud scans.

  9. How the Path Matching Algorithm works

    main

    ProxSave uses a specific algorithm to match archive entries to category paths.

    1. Normalization: Every entry name is checked. If it does not start with ./, the prefix ./ is prepended.
    2. Matching Logic:
      • Exact Match: If the entry name matches a category path exactly, it is a match.
      • Prefix Match: If the category path ends with /, the entry is a match if the entry starts with that path.
      • Parent Match: If the entry does not end in /, the entry is a match if it equals the category path without the trailing slash.

    Examples:

    Archive EntryCategory PathResultReason
    ./etc/network/interfaces./etc/network/✅ MatchPrefix match
    ./etc/hostname./etc/network/❌ No matchNo prefix/exact match
    etc/hostname./etc/hostname✅ MatchNormalized to ./etc/hostname
    ./var/lib/pve-cluster/config.db./var/lib/pve-cluster/✅ MatchPrefix match

  10. Understand Proxmox VE Cluster Architecture for Recovery

    main

    To perform successful cluster recovery with proxsave, you must understand the relationship between the Proxmox VE filesystem and its underlying database.

    The Cluster Stack

    1. Application Layer: Services like pveproxy and pvedaemon interact with the cluster.
    2. Cluster Filesystem Layer (/etc/pve): This is a FUSE mount, not a standard directory. It provides a view of the cluster configuration (e.g., storage.cfg, datacenter.cfg, VM configs).
    3. pmxcfs (Cluster Filesystem Daemon): The daemon that manages the FUSE mount and translates filesystem operations into database queries.
    4. Database Layer (config.db): The actual source of truth. It is an SQLite database located at /var/lib/pve-cluster/config.db. This is the file that proxsave restores.
    5. Communication Layer (Corosync): Handles cluster communication, quorum, and the synchronization of config.db across all nodes.

    Critical Warning: /etc/pve is not a real directory

    Because /etc/pve is a FUSE mount, you cannot use standard commands like cp to restore configuration files. Any files copied directly into /etc/pve will be lost when the pmxcfs daemon restarts.

    # This will NOT persist after a restart:
cp backup/etc/pve/storage.cfg /etc/pve/storage.cfg

  11. Use SAFE vs RECOVERY mode for Cluster restores

    main

    When restoring a backup from a Proxmox VE cluster, ProxSave detects the ClusterMode: "cluster" flag in the manifest and prompts the user to choose between two strategies:

    RECOVERY Mode

    • Behavior: Direct database restoration.
    • Use Case: When you want to overwrite the existing cluster state immediately.

    SAFE Mode

    • Behavior: Instead of restoring directly to the system, the pve_cluster category is redirected to ExportOnly. Files are extracted to an export directory rather than /etc/pve.
    • Workflow: After extraction, the user can use runSafeClusterApply() to selectively apply configurations via the pvesh API.
    • API-based Application: SAFE mode allows applying:
      1. VM/CT Configs: Scans and applies via pvesh set /nodes/<node>/<type>/<vmid>/config.
      2. Storage Config: Parses storage.cfg and applies via pvesh set /cluster/storage/<id>.
      3. Datacenter Config: Applies via pvesh set /cluster/config -conf <file>.

  12. How Category Type Filtering works

    main

    ProxSave filters available categories based on the detected system type (PVE, PBS, DUAL, or Unknown) and the selected restore mode.

    1. System Type Filtering

    • PVE: Includes CategoryTypePVE and CategoryTypeCommon.
    • PBS: Includes CategoryTypePBS and CategoryTypeCommon.
    • DUAL: Includes CategoryTypePVE, CategoryTypePBS, and CategoryTypeCommon.
    • Unknown: Includes CategoryTypeCommon only.

    2. Restore Mode Filtering

    • Full / Storage / Base Modes: Automatically removes categories where ExportOnly = true.
    • Custom Mode: Keeps all categories, allowing the user to choose specifically what to restore.

    3. Availability Check

    After filtering, ProxSave verifies if the categories are actually present in the archive using IsAvailable before presenting the final list.