DOCSEARCHbyAlgolia
Submit Docs

Datashelter Documentation

website·Indexed 32 days ago

https://docs.datashelter.tech

Documentation for Datashelter and its CLI tool, Snaper. Includes guides on system and database backups, storage usage calculation, retention and downsampling policies, 2FA activation, and troubleshooting for MongoDB and S3-compatible endpoints.

Tokens
16.8K
Snippets
74
Records
154
Agent score
53%

What's inside Datashelter

  1. Overview of Datashelter backup solution

    Datashelter is an integrated backup solution for Linux servers designed to automate the backup of files and databases to S3-compatible storage. It provides built-in encryption, compression, deduplication, and scheduling to ensure backups are lightweight and secure.

  2. Overview of Datashelter Backup as a Service

    Datashelter is a Backup as a Service (BaaS) solution designed to replace manual backup management (e.g., custom rsync or restic scripts). It provides a managed environment that handles backup integrity, monitoring, alerting, and storage orchestration, allowing for setup in approximately 5 minutes.

  3. Understand Datashelter infrastructure and security features

    Datashelter is built on the S3 protocol and utilizes a custom S3 gateway to provide the following infrastructure features:

    • Data Tampering Protection: Backups are immutable (cannot be deleted or modified) and users are granted read-only access, ensuring security even if S3 credentials are compromised.
    • Strong Segmentation: Every server created in Datashelter is isolated with its own dedicated bucket and credentials to prevent cross-server impact in the event of a compromise.
    • Automated Alerts: Continuous monitoring notifies users via email if backups are not received.
    • Retention Management: Supports configurable downsampling rules for database dumps (e.g., daily, weekly, and monthly rotations). For file backups, deduplication allows indexes to be kept for 1 year.

  4. Understand Snaper agent security and command execution

    The Snaper agent cannot run arbitrary shell commands, expose a shell, or provide SSH sessions. It operates on a pull-model, reading instructions from a private Datashelter S3 bucket and executing only a hard-coded set of predefined actions. Security is enforced by:

    1. No Inbound Ports: The agent only makes outbound calls to the S3 bucket and Datashelter API.
    2. Hard-coded Command Set: Only specific command types are recognized; all other payloads are ignored.
    3. Signed Commands: Every command is signed and scoped to the specific server identity.
    4. Auditability: Activity can be monitored via logs in /tmp/snaper and the snaper agent status command.

  5. Understand and use degressive retention policies for backups

    Degressive retention policies, also known as downsampling, reduce the number of stored backup versions over time. This approach keeps a high frequency of recent backups and a lower frequency of older backups to optimize storage costs. This is specifically recommended for database backups because deduplication is often ineffective on database dumps, even with high compression ratios.

  6. Secure data transmission with Snaper backup agent

    The Snaper backup agent ensures data security during transit using three primary mechanisms:

    1. S3 API Communications: All interactions use the standard S3 protocol signed with S3 Signature v4. Every call is encrypted via TLS (HTTPS), protecting against tampering and replay attacks.
    2. Push-Only Model: Servers push backups out to Datashelter; Datashelter never initiates inbound connections. This eliminates the need for inbound ports and works behind NAT/firewalls.
    3. No SSH Access: Datashelter does not require direct SSH access, service accounts, or bastion hosts on your servers.
    # Network control recommendation: 
# Restrict outbound rules to allow only HTTPS connections to s3.datashelter.cloud

  7. Understand why Snaper uses the s3.datashelter.cloud proxy

    Snaper uses a dedicated S3 proxy at s3.datashelter.cloud to provide immutable storage (WORM - Write Once Read Many) features. All communication is performed via HTTPS calls using the S3 Signature v4 authentication algorithm. The proxy prevents irreversible data alteration through two primary mechanisms:

    1. Blocking Deletions: The proxy accepts DELETE requests at the protocol level but does not perform the deletion in the actual storage.
    2. Versioning Overwrites: When a PUT request attempts to overwrite an existing key, the proxy creates a new version of the object instead of replacing the previous one.

    These safeguards ensure data integrity and non-alteration even in the event that the server's S3 credentials are compromised.

  8. Understand Datashelter backup security during account compromise

    If a Datashelter account is compromised, the attacker cannot access the contents of the backups because all data is encrypted locally on the user's server before transmission. The attacker only has access to server/backup names, retention policy settings, and the ability to attempt deletions. To mitigate this risk, users should enable two-factor authentication (2FA).

  9. Manually install Snaper CLI binaries

    If the automatic installer cannot be used, you can manually download the binary for your OS and architecture. After downloading, make the file executable and move it to a directory on your PATH (e.g., /usr/local/bin or $HOME/bin).

    Supported Platforms:

    • Linux (amd64, arm64)
    • macOS/Darwin (amd64, arm64)
    • FreeBSD (amd64, arm64)
    • Windows

    Note: The -k flag in the curl commands disables strict TLS verification; remove it if strict verification is preferred.

    # Example for Linux amd64
curl -LO -k https://dl.datashelter.cloud/release/latest/bin/linux/amd64/snaper \
&& chmod +x ./snaper \
&& sudo mv ./snaper /usr/local/bin/snaper \
&& snaper --version

# Example for macOS (Darwin) arm64
curl -LO -k https://dl.datashelter.cloud/release/latest/bin/darwin/arm64/snaper \
&& chmod +x ./snaper \
&& sudo mv ./snaper /usr/local/bin/snaper \
&& snaper --version

  10. Understand Datashelter immutable backup storage

    Datashelter provides native immutable storage for backups, ensuring that data cannot be deleted or modified, even in the event of compromised S3 credentials. This immutability is enforced logically via the s3.datashelter.cloud proxy used to interact with S3 buckets, regardless of whether the bucket is hosted by Datashelter or a third-party provider. Users are granted read-only access to these backups.

  11. Verify OS and architecture compatibility for Snaper

    Snaper is compatible with Linux, macOS, Windows, and FreeBSD. It is designed to run on various environments including NAS devices, personal computers, and production servers. It supports both amd64 and arm64 architectures.

  12. Manage backup recurrence with snaper crontab management

    Introduced in July 2024, snaper includes a crontab management feature that allows users to set a recurrence for backups (e.g., daily database dumps at a specific time) directly via the panel. Snaper implements this by managing its own dedicated section within the server's crontab list, adding a unique crontab entry for every backup with a defined recurrence.
    # SNAPER MANAGED CRONTABS
5 1 * * * /usr/local/bin/snaper backup files --name lingering-moon >> /tmp/snaper.log 2>&1
# END OF SNAPER MANAGED CRONTABS