K8up Documentation

repository·master·Indexed 21 days ago

https://github.com/k8up-io/k8up

K8up is a Kubernetes backup operator based on Restic that automates PVC and application backups using custom resources such as `schedule` and `credentials`. It manages backups on Kubernetes and OpenShift clusters via an operator module and an ephemeral Restic module. The documentation covers installation via Helm, configuration of operator behavior, RBAC and security settings, and detailed upgrade paths from v0 through v4.

Tokens
45.4K
Snippets
116
Records
176
Agent score
77%

What's inside K8up

  1. Overview of K8up Backup Operator

    master

    K8up is a Kubernetes backup operator built on Restic. It manages PVC (Persistent Volume Claim) and application backups on Kubernetes or OpenShift clusters.

    To perform backups, you primarily need to create two objects in the target namespace:

    1. A schedule object to define backup frequency.
    2. A credentials object to manage access to your backup storage.

    K8up also provides a Prometheus endpoint for monitoring backup activities.

  2. Overview of K8up capabilities

    master

    K8up is a Kubernetes Operator (compatible with Kubernetes and OpenShift) designed for managing backups and archives. It is built on top of Restic and supports S3-compatible storage (e.g., Amazon S3, Minio).

    Key capabilities include:

    • PVC Backups: Back up all PVCs marked as ReadWriteMany, ReadWriteOnce, or those with a specific label.
    • On-demand Backups: Perform individual backups manually.
    • Scheduled Backups: Execute backups on a regular basis.
    • Scheduled Archivals: Execute long-interval tasks like archiving to AWS Glacier.
    • Application-Aware Backups: Capture the output of any tool that writes to stdout.
    • Integrity Checks: Verify the integrity of the backup repository.
    • Pruning: Remove old backups from a repository.
    • Restoration: Restore backups using the k8up CLI tool.
  3. Configure Mutual TLS (mTLS) for Archive objects

    master

    For environments requiring mutual TLS (mTLS) for client authentication, you must provide the CA certificate, the client certificate, and the client key. This can be configured via tlsOptions or environment variables.

    Using tlsOptions

    In the spec, define the following under tlsOptions:

    • caCert: Path to the CA certificate.
    • clientCert: Path to the client certificate.
    • clientKey: Path to the client private key.

    Using env

    Use a ConfigMap to provide the following environment variables:

    • CA_CERT_FILE: Path to the CA certificate.
    • CLIENT_CERT_FILE: Path to the client certificate.
    • CLIENT_KEY_FILE: Path to the client private key.
    • RESTORE_CA_CERT_FILE: Path to the CA certificate for restoration.
    • RESTORE_CLIENT_CERT_FILE: Path to the client certificate for restoration.
    • RESTORE_CLIENT_KEY_FILE: Path to the client private key for restoration.
    apiVersion: k8up.io/v1
    kind: Archive
    metadata:
      name: archive-test
    spec:
      backend:
        s3: {}
      tlsOptions:
        caCert: /mnt/tls/ca.crt
        clientCert: /mnt/tls/tls.crt
        clientKey: /mnt/tls/tls.key
      volumeMounts:
        - name: client-tls
          mountPath: /mnt/tls/
      restoreMethod:
        s3: {}
      tlsOptions:
        caCert: /mnt/tls/ca.crt
        clientCert: /mnt/tls/tls.crt
        clientKey: /mnt/tls/tls.key
      podSecurityContext:
        fsGroup: 1000
        runAsUser: 1000
      volumes:
        - name: client-tls
          secret:
            secretName: client-tls
            defaultMode: 420
  4. How K8up architecture works

    master

    K8up is composed of two primary components that work together to manage Kubernetes data backups:

    1. K8up Operator: A cluster-wide operator responsible for managing Backup and Schedule resources. It monitors schedules, identifies matching PersistentVolumeClaims (PVCs), and orchestrates the creation of backup jobs.
    2. K8up Restic: A wrapper around the restic binary. It is executed within a Kubernetes Job to perform the actual data movement from the mounted PVCs to the configured backup endpoint.

    The Backup Workflow: When a Schedule triggers an action (like a backup), the operator creates a Backup custom resource. This triggers the creation of a Kubernetes Job containing a Pod. The operator mounts the target PVC(s) into the Pod at /data/<pvcname>. The Pod then runs k8up restic, which uses the mounted data to perform the backup.

  5. How K8up is architected

    master

    K8up is composed of two main functional modules:

    • Operator module: Runs continuously within the Kubernetes cluster and manages various reconciliation loops (e.g., watching for new backup requests).
    • Restic module: Acts as the interface to the restic binary. This module is invoked whenever a Backup or Restore custom resource is instantiated. The process for this module is ephemeral: it starts to perform the specific job and terminates once the task is complete.
  6. Configure a Schedule with K8up

    master

    The Schedule CRD allows you to automate all other K8up operations (Backup, Restore, Archive, Check, Prune) on a recurring basis. You define the timing for each operation type and the storage backend to be used.

    Key components of a Schedule include:

    • archive: Configuration for archival jobs.
    • backend: The storage destination (e.g., S3, Azure, GCS).
    • check: Configuration for repository integrity checks.
    • backup: Configuration for the backup jobs, including cron schedules and history limits.
    • prune: Configuration for snapshot retention and cleanup.

    Note: failedJobsHistoryLimit and successfulJobsHistoryLimit control how many finished job/pod objects are kept after cleanup. They default to 3.

    apiVersion: k8up.io/v1
    kind: Schedule
    metadata:
      name: schedule-test
    spec:
      backend:
        repoPasswordSecretRef:
          name: backup-repo
          key: password
        s3:
          endpoint: http://10.144.1.224:9000
          bucket: k8up
          accessKeyIDSecretRef:
            name: backup-credentials
            key: username
          secretAccessKeySecretRef:
            name: backup-credentials
            key: password
      archive:
        schedule: '0 * * * *'
        restoreMethod:
          s3:
            endpoint: http://10.144.1.224:9000
            bucket: restoremini
            accessKeyIDSecretRef:
              name: backup-credentials
              key: username
            secretAccessKeySecretRef:
              name: backup-credentials
              key: password
      backup:
        schedule: '* * * * *'
        failedJobsHistoryLimit: 4
        successfulJobsHistoryLimit: 0
        promURL: http://10.144.1.224:9000
      check:
        schedule: '*/5 * * * *'
        promURL: http://10.144.1.224:9000
      prune:
        schedule: '*/2 * * * *'
        retention:
          keepLast: 5
          keepDaily: 14
  7. Understand Application-Aware Backups

    master

    Application-Aware Backups allow you to perform consistent backups of complex applications like databases by executing a command within your existing application Pod.

    How it works: K8up uses an annotation on your Pod to define a backup command. During a backup, K8up executes this command inside the Pod (similar to kubectl exec POD -- COMMAND ARGS) and collects everything written to STDOUT. The collected STDOUT stream is then stored as a file in your configured backup storage.

    When to use it:

    • Databases: To get a consistent view of data (e.g., using pg_dump for PostgreSQL) instead of potentially corrupting files by reading them while they are being modified.
    • Internal Access: When the backup process needs access to the same internal/external endpoints and network context as the application Pod.

    Limitations:

    • Efficiency: Data is transferred via STDOUT through the Kubernetes API, which is less efficient than direct volume access.
    • Tooling Requirements: The container image used by your Pod must contain the necessary binaries to execute the backup command (e.g., it will not work with 'distroless' images unless you use the PreBackupPod method).
  8. Understand PreBackupPod

    master

    The PreBackupPod method is a flexible hybrid that uses a dedicated, temporary Pod to perform backups, rather than running commands inside your existing application Pod.

    How it works: K8up creates a special Pod for every backup run based on a definition you provide. This Pod can have its own custom image, Secrets, and configuration. Once the Pod starts, K8up executes a specified backupCommand inside it. After the command completes, K8up removes the Pod.

    When to use it:

    • Distroless/Minimal Images: When your application container lacks the tools (like pg_dump) needed to perform a backup.
    • Custom Tooling: When you want to use a specific, heavy image containing all necessary backup utilities without bloating your application image.
    • External Connectivity: When you need to connect to managed databases or external services using specific credentials stored in Kubernetes Secrets.

    Limitations:

    • Isolation: The PreBackupPod runs in its own context and cannot access services that are internal to your application Pod (e.g., it cannot reach a service listening on localhost inside your app container).
    • Maintenance: You must keep the PreBackupPod definition in sync with your application's requirements (e.g., if you update your database version, you may need to update the PreBackupPod image).
  9. How pod resource precedence works in K8up

    master

    K8up manages pod resources (CPU and memory) using a three-level hierarchy. Each level takes precedence over the one before it. This allows administrators to set global defaults while allowing specific jobs to override them.

    Precedence Order (Highest to Lowest):

    1. Specific Object Specs: The resources field defined directly within a Backup, Prune, Check, Archive, or Restore spec inside a Schedule.
    2. Schedule Templates: The resourceRequirementsTemplate field defined in a Schedule object.
    3. Global Defaults: Environment variables set on the K8up operator.

    Important Behaviors:

    • Non-persistence: The final merged resource configuration is computed at runtime and is not persisted in the Schedule object. This allows administrators to update global defaults without modifying individual Schedule resources.
    • Manual Objects: If you create standalone Backup objects (instead of letting a Schedule generate them), neither the global defaults nor the resourceRequirementsTemplate from any Schedule will be applied.
  10. Define a Backup or BackupSchedule

    master

    A Backup (or a BackupSchedule for automated backups) requires a BackupSpec. This spec defines how the backup job should run, including:

    • backend: The storage destination.
    • resources: Compute resource requirements (CPU/Memory).
    • podSecurityContext: Security settings for the job pod.
    • podConfigRef: An advanced option to reference a PodConfig for custom pod specifications (takes precedence over resources and podSecurityContext).
    • volumes: Volumes to mount in the job pod.
    • activeDeadlineSeconds: Maximum duration for the job.
    • labelSelectors: Filters to determine which PVCs and PreBackupPods are included in the backup.
    • tags: Restic tags to apply to the backup.
    ### BackupSpec Fields
    | Field | Description
    | *`backend`* | Backend containing the restic repo destination
    | *`resources`* | Compute resource requirements
    | *`podSecurityContext`* | Security context for execution
    | *`podConfigRef`* | Reference to a PodConfig (advanced use-case)
    | *`volumes`* | List of volumes to mount
    | *`activeDeadlineSeconds`* | Max duration in seconds
    | *`failedJobsHistoryLimit`* | Number of failed jobs to keep
    | *`successfulJobsHistoryLimit`* | Number of successful jobs to keep
    | *`promURL`* | Prometheus push URL for metrics
    | *`clusterName`* | Cluster name for metric grouping
    | *`statsURL`* | URL for posting snapshot metrics
    | *`tags`* | Restic tags
    | *`labelSelectors`* | Selectors to filter PVCs and PreBackupPods