DOCSEARCHbyAlgolia
Submit Docs

Altinity Backup for ClickHouse

repository·master·Indexed 23 days ago

https://github.com/altinity/clickhouse-backup

A utility for creating and restoring backups of ClickHouse databases. It leverages ClickHouse's file system snapshots via hard links and supports various cloud and non-cloud storage types. The tool must be run on the same host, Kubernetes Pod, or neighbor container as the clickhouse-server to perform full data backups and restores.

Tokens
32K
Snippets
53
Records
137
Agent score
82%

What's inside altinity-clickhouse-backup

  1. What is the ACVP Wrapper and its compliance scope?

    master

    The ACVP wrapper is a reproducibility harness used for ACVP (Automated Cryptographic Validation Program) expected-output tests. It is used by clickhouse-backup-acvp and clickhouse-backup acvp.

    Important Distinctions:

    • It is not a CMVP certificate artifact itself.
    • The actual compliance-relevant cryptographic implementation is Go's native crypto/internal/fips140/... module when executed with the environment variable GODEBUG=fips140=on (or only).
    • The tracked configuration pkg/acvpwrapper/acvp_test_fips140v1.26.public.config.json is a derivative of Go's v1.26 ACVP config, specifically excluding ML-KEM and ML-DSA vector suites to ensure testing remains strictly within publicly exposed Go crypto APIs.

  2. Important: Do not run clickhouse-backup remotely

    master
    The clickhouse-backup tool must be executed on the same host where the ClickHouse server is running. It is designed to interact with the local filesystem and ClickHouse instance directly; running it from a remote machine will not work as intended.

  3. Deployment requirement: Run clickhouse-backup on the same host as ClickHouse

    master

    To perform full data backups and restores, clickhouse-backup must have direct access to the ClickHouse data files located in /var/lib/clickhouse.

    **You must run clickhouse-backup on:

    • The same host as clickhouse-server
    • The same Kubernetes Pod
    • A neighbor container on the same host where clickhouse-server is running

    WARNING: If you connect to a remote clickhouse-server host, you can only backup and restore the schema, not the actual data.

  4. How incremental backups work with remote storage

    master

    Incremental backups calculate only the changes during upload or create_remote commands. The efficiency of this process depends on the use_embedded_backup_restore configuration:

    • use_embedded_backup_restore: false: Incremental calculation happens at the table parts level.
    • use_embedded_backup_restore: true: Incremental calculation happens via file-level checksums, which is more effective.

    Key Requirements and Behaviors:

    • The backup specified by the --diff-from parameter must be present as a local backup. Use clickhouse-backup list to verify.
    • During Upload: The base_backup is added to the current backup's metadata as required_backup in backup_name/metadata.json. Data parts existing in the base backup are marked with a required flag in backup_name/metadata/database/table.json and are skipped during upload.
    • During Download: If a backup has a required_backup link, parts marked as required are downloaded to local storage after the non-required parts are complete. This process is recursive for backup chains.

    Optimization Tip: To reduce the size of increments, increase the number of rows per INSERT query and avoid frequent table data mutations, as background merges affect increment size.

  5. Format partition identifiers for CLI commands

    master

    When using --partitions with download, restore, or other commands, the format depends on the PARTITION BY clause of your ClickHouse tables. Refer to system.parts for exact values.

    Partition TypeFormat Example
    Numeric (non-hashed)--partitions=part1,part2
    Hashed Strings--partitions=('val1'),('val2')
    Tuples (Multiple fields)--partitions=(1,'str','2023-01-01'),(2,'str2','2023-01-02')
    Per-table partitions--partitions=db.table1:part1,part2 --partitions=db.table?:*

    Note: Always use single quotes for String and Date/DateTime related types.

  6. Implement custom remote storage using go-templates

    master

    You can implement custom storage providers by defining custom commands using the go-template language.

    Available template variables include:

    • {{ .cfg.* }}: Access configuration values.
    • {{ .backupName }}: The name of the current backup.
    • {{ .diffFromRemote }}: Information regarding the difference from the remote storage.

    Requirements for list_command: Your custom list_command must return JSON that is compatible with the metadata.BackupMetadata type, specifically using the ClickHouse JSONEachRow format.

  7. Override configuration with environment variables

    master
    You can override any parameter defined in the configuration file using environment variables. Environment variable names must be UPPERCASE. The mapping between the config key and the environment variable is indicated in the configuration documentation by a # following the key (e.g., remote_storage is overridden by REMOTE_STORAGE).

  8. How the watch command works

    master

    The watch command automates periodic backups. Its lifecycle is as follows:

    1. It starts by executing a create_remote+delete sequence to create a full backup.
    2. It waits for the duration specified by watch-interval.
    3. It then executes the create_remote+delete sequence again.
    4. The backup type is determined by the full-interval: if the time since the last full backup exceeds full-interval, it performs a full backup; otherwise, it performs an incremental backup.

  9. How clickhouse-backup works

    master
    The tool creates backups by interacting with the ClickHouse server and the local filesystem. It typically involves a two-step process for remote storage: first, creating a local backup, and second, uploading that backup to a remote storage provider (like S3).

  10. How backups_to_keep_remote works

    master

    The backups_to_keep_remote mechanism manages remote backup retention by:

    1. Fetching the list of remote backups.
    2. Selecting the oldest backups to delete so that only the number of backups specified by backups_to_keep_remote remains.
    3. Dependency Safety: Before deleting a remote backup, the system checks for dependencies. A dead remote backup will not be deleted if any live backup has a direct or recursive reference to it.

  11. How clickhouse-backup works (Freeze and Attach)

    master

    The tool leverages the immutability of ClickHouse data files:

    1. Backup: clickhouse-backup executes the ALTER TABLE ... FREEZE query. This creates file system hard links to the existing clickhouse-server data parts, allowing for a consistent snapshot without stopping the server.
    2. Restore: clickhouse-backup copies the hard links to the detached folder and executes the ALTER TABLE ... ATTACH PART query for each data part and table in the backup.

  12. Perform a one-time data restore using Kubernetes Job

    master

    For a one-time restore of a sharded cluster, use a Kubernetes Job. The process involves two distinct phases to ensure schema consistency across the cluster and data consistency on specific replicas.

    Restore Workflow:

    1. Schema Restore: Run restore_remote --schema --rm <BACKUP_NAME> on all replicas in the cluster. This ensures the table structures are identical everywhere.
    2. Data Restore: Run restore_remote --data <BACKUP_NAME> on only one replica per shard. This populates the data without creating duplicate data across replicas.
    3. Cleanup: After a successful restore, run delete local <BACKUP_NAME> to remove temporary files.

    Key Environment Variables for the Restore Job:

    • CLICKHOUSE_SCHEMA_RESTORE_SERVICES: A comma-separated list of all service hostnames for schema restoration.
    • CLICKHOUSE_DATA_RESTORE_SERVICES: A comma-separated list of hostnames (one per shard) for data restoration.
    • BACKUP_USER / BACKUP_PASSWORD: Credentials for the ClickHouse cluster.
    apiVersion: batch/v1
kind: Job
metadata:
  name: clickhouse-backup-restore
spec:
  template:
    spec:
      containers:
        - name: clickhouse-backup-restore
          image: clickhouse/clickhouse-client:latest
          env:
            - name: CLICKHOUSE_SCHEMA_RESTORE_SERVICES
              value: "chi-test-backups-default-0-0,chi-test-backups-default-0-1,chi-test-backups-default-1-0,chi-test-backups-default-1-1"
            - name: CLICKHOUSE_DATA_RESTORE_SERVICES
              value: "chi-test-backups-default-0-0,chi-test-backups-default-1-0"
            - name: CLICKHOUSE_PORT
              value: "9000"
            - name: BACKUP_USER
              value: "backup"
            - name: BACKUP_PASSWORD
              value: "backup_password"
          command:
            - bash
            - -ec
            - | 
              # ... (bash script to execute restore_remote commands via system.backup_actions)