DOCSEARCHbyAlgolia
Submit Docs

Velero

repository·main·Indexed 11 days ago

https://github.com/vmware-tanzu/velero

A Kubernetes backup and restore tool designed to protect cluster resources and persistent volumes, enabling cluster migration and environment replication. It consists of a Velero Server running inside the cluster and a Velero CLI for local interaction. Supports public cloud platforms and on-premises environments, with specific compatibility matrices for Kubernetes versions across Velero releases 1.14 through 1.18.

Tokens
174.1K
Snippets
517
Records
759
Agent score
94%

What's inside Velero

  1. Overview of Velero

    main

    Velero (formerly Heptio Ark) is a tool for backing up and restoring Kubernetes cluster resources and persistent volumes. It can be used with public cloud platforms or on-premises environments.

    Core Capabilities:

    • Backup and Restore: Take backups of your cluster and restore them in case of resource loss.
    • Migration: Move cluster resources from one cluster to another.
    • Replication: Replicate production clusters to development and testing environments.

    Architecture: Velero operates using two main components:

    1. Velero Server: Runs inside your Kubernetes cluster.
    2. Velero CLI: A command-line client that runs locally on your machine to interact with the server.

  2. Rapid iterative Velero development with Tilt

    main

    Use Tilt to create a simplified development workflow for Velero. This setup allows for continuous deployment of the Velero server, provider plugins, and the node-agent daemonset. Tilt automates the process by:

    1. Deploying Kubernetes resources (CRDs, Velero deployment).
    2. Building local binaries for Velero and provider plugins as local_resource.
    3. Using docker_build to live-update binaries into containers and trigger restarts.

    Configuration files are expected to be located in velero/tilt-resources.

  3. What is Velero Generic Data Path (VGDP) concurrency?

    main

    Velero Generic Data Path (VGDP) refers to the collection of modules (including uploaders and backup repository connections) used to perform data transfers, such as PodVolume backup/restore and Volume Snapshot Data Movement.

    Because VGDP instances can run concurrently and consume significant CPU, memory, and network bandwidth, Velero provides a mechanism to limit how many VGDP instances run on a single node at once. This prevents a single node from becoming overwhelmed by multiple simultaneous backup or restore operations.

  4. What is CSI Snapshot Data Movement

    main

    CSI Snapshot Data Movement is a feature designed to move CSI snapshot data from a source cluster to a durable backup storage location. Unlike standard CSI snapshot backups which only take the snapshot, this feature uses 'data movers' to access the snapshot data (at the block level, fully or incrementally) and upload it to a pre-defined backup storage (e.g., S3-compatible object storage).

    Key Benefits:

    • On-premises: Moves snapshots from storage that doesn't support durable snapshots to low-cost, large-scale object storage.
    • Multi-cloud: Allows backing up volume snapshots from one cloud provider and restoring them to another.
    • Consistency: Preferred over File System Backup because it reads from the block-level snapshot rather than the live PV, ensuring better point-in-time consistency.

    How it works:

    1. Velero takes a CSI snapshot.
    2. A data mover (like the built-in Kopia uploader) reads the snapshot data.
    3. Data is written to the backup storage.
    4. Once complete, Velero removes the CSI snapshot to release storage space.

  5. How Velero plugins work

    main

    Velero uses a plugin architecture that allows adding custom functionality without modifying the core binary.

    Workflow:

    1. Create a custom binary containing implementations of Velero's plugin kinds and boilerplate code to expose them.
    2. Add this binary to a container image.
    3. Use that image as an init container for the Velero server pod.
    4. The init container copies the binary into a shared emptyDir volume, which the Velero server then accesses.

    A single binary can implement multiple plugin types.

  7. Configure cache volumes for Data Mover pods

    main
    To prevent data mover pod failures due to ephemeral disk limits or node disk constraints during restore operations, you can now configure cache volumes. This applies to both CSI snapshot data movement and fs-backup. Combining appropriately sized cache volumes with backup repository cache limit configuration can improve overall restore throughput.

  9. Troubleshoot LoadBalancer Service restores

    main

    When restoring Kubernetes Services of type=LoadBalancer, the Service UID changes. Because cloud providers often generate the cloud resource name based on this UID, the restored service may result in a different cloud load balancer name.

    Solutions:

    1. Update your application's DNS CNAME pointer to the new cloud load balancer DNS name.
    2. If your cloud provider supports it, use the Service's spec.loadBalancerIP field to maintain connection stability.

  10. How CSI VolumeSnapshot cleanup works during Backup

    main

    During a Velero backup, the system is designed to clean up backup-generated VolumeSnapshot (VS) and VolumeSnapshotContent (VSC) resources once the backup phase reaches Finalizing or FinalizingPartiallyFailed.

    To ensure the underlying snapshots in the storage provider are not accidentally deleted when the Kubernetes resources are removed, the cleanup process follows these steps:

    1. It identifies the VolumeSnapshotContent (VSC) bound to the VolumeSnapshot (VS).
    2. It patches the DeletionPolicy of the VSC to Retain.
    3. It deletes the VSC.
    4. It deletes the VS.

    This approach simplifies the logic by removing the need to recreate static VSCs pointing to non-existent VS resources.

    if backup.Status.Phase == velerov1api.BackupPhaseFinalizing ||
	backup.Status.Phase == velerov1api.BackupPhaseFinalizingPartiallyFailed {
	// ... logic to patch VSC DeletionPolicy to Retain and then delete VSC and VS
}

  11. Configure volume snapshot providers for persistent volumes

    main

    To back up persistent volume data, you must select a volume backup solution:

    1. Native Snapshot Plugins: If your storage platform provides a Velero plugin (e.g., Portworx), use it to obtain native snapshots as part of your backups.
    2. File System Backup (FSB): If no native snapshot plugin is available for your storage platform, use Velero's File System Backup feature, which provides a platform-agnostic, file-level backup solution for volume data.

  12. Configure Velero API types via JSON/YAML

    main

    While most Velero operations can be performed via the velero CLI, certain API types require direct JSON or YAML configuration to access their full functionality. This is particularly true for defining complex behaviors like hooks. The following API types can be configured using manifests:

    • Backup: Defines a backup operation.
    • Schedule: Defines a recurring backup schedule.
    • BackupStorageLocation: Configures where backups are stored (e.g., S3, Azure Blob).
    • VolumeSnapshotLocation: Configures where volume snapshots are stored.

    When using the CLI for these resources, you may be limited to a subset of available fields. For advanced configurations, create a YAML manifest and apply it using velero restore create --from-backup ... or similar resource-specific commands (depending on the resource type).