DOCSEARCHbyAlgolia
Submit Docs

Containerized Data Importer (CDI)

repository·main·Indexed 19 days ago

https://github.com/kubevirt/containerized-data-importer

A Kubernetes extension that provides a declarative way to populate PersistentVolumeClaims (PVCs) with Virtual Machine disk images or other data. Primarily used to automate disk creation for KubeVirt VMs and general volume initialization.

Tokens
52.4K
Snippets
149
Records
227
Agent score
67%

What's inside containerized-data-importer

  1. What is a DataVolume?

    main

    A DataVolume is a Custom Resource Definition (CRD) provided by CDI that acts as an abstraction on top of a standard Kubernetes PersistentVolumeClaim (PVC).

    While you can use PVCs directly with CDI, using DataVolumes is the preferred method because they:

    • Automate the creation and population of a PVC with data.
    • Offer a stable API.
    • Provide full functionality and better integration with KubeVirt.

    DataVolumes allow you to declaratively define how a volume should be populated from various sources such as URLs, container registries, other PVCs, or client uploads.

  2. What are Data Volumes (DV)?

    main

    A DataVolume (DV) is a Kubernetes abstraction built on top of Persistent Volume Claims (PVC) and the Containerized Data Importer (CDI).

    Its primary purpose is to monitor and orchestrate the import, upload, or cloning of data into a PVC. Unlike relying on raw CDI state annotations, DataVolumes provide a versioned API, allowing projects like KubeVirt to integrate with a stable interface that guarantees consistent behavior across versions.

  3. Understand Data Volume cloning methods

    main

    Containerized Data Importer (CDI) supports three distinct cloning methods for DataVolumes, which are selected based on the capabilities of your storage provider and the configuration of your PVCs. The goal is to use the most efficient method available:

    1. CSI Volume Cloning: The most efficient method. It leverages the CSI driver's native volume cloning capabilities.
    2. Smart Cloning: A highly efficient method that uses CSI snapshots. It requires a Snapshot Class to be associated with your Storage Class.
    3. Host-Assisted Cloning (cloneType: copy): The fallback method. It is the least efficient as it uses a source pod and a target pod to manually copy data from the source volume to the target volume. This occurs automatically if the prerequisites for CSI or Smart cloning are not met.

    Note that cloning can be combined with namespace transfer and size expansion.

  4. How Smart-Cloning works for Data Volumes

    main

    Smart-Cloning is a performance optimization in CDI that uses Kubernetes volume snapshots to clone PersistentVolumeClaims (PVCs) instead of using the slower host-assisted streaming method.

    When a DataVolume is created with a PVC source, CDI attempts to use Smart-Cloning if the following conditions are met:

    1. The source and target PVCs are in the same StorageClass.
    2. There is a VolumeSnapshotClass associated with that StorageClass.
    3. The source PVC is not currently mounted by any Pod.

    If these conditions are met, CDI creates a snapshot of the source, creates the new PVC from that snapshot, deletes the snapshot, and expands the new PVC if a larger size was requested. If these conditions are not met, CDI falls back to a slower host-assisted clone.

    Note: Some CSI drivers require the new PVC size to exactly match the size of the PVC from which the snapshot was created when restoring.

  5. How CDI determines scratch space storage classes

    main

    Containerized Data Importer (CDI) uses temporary scratch space to process data before writing it to the target PersistentVolumeClaim (PVC). To ensure successful completion, CDI creates scratch space equal in size to the DataVolume (DV).

    CDI follows this priority order to select a storage class for scratch space:

    1. CDI Config: It checks the scratchSpaceStorageClass field in the CDI configuration. If this field exists and matches a valid storage class in the cluster, it is used.
    2. DataVolume PVC: If the scratchSpaceStorageClass field is blank, CDI uses the storage class of the PVC backing the DataVolume that initiated the operation.

    If neither is available, operations requiring scratch space will fail, though operations that do not require scratch space will continue to function.

  6. Understand DataVolume status phases

    main

    You can monitor the progress of a DataVolume operation by checking its status. The following phases are possible:

    • Blank: No status available.
    • Pending: Operation is pending but not yet scheduled.
    • WaitForFirstConsumer: The associated PVC is Pending and the storage uses WaitForFirstConsumer binding mode; it is waiting for a Pod to consume it.
    • PVCBound: The associated PVC has been bound.
    • ImportScheduled, CloneScheduled, or UploadScheduled: The operation has been scheduled.
    • ImportInProgress, CloneInProgress, or UploadInProgress: The operation is currently running.
    • SnapshotForSmartClone or SmartClonePVCInProgress: A Smart-Cloning operation is in progress.
    • CSICloneInProgress: A CSI Volume Clone operation is in progress.
    • CloneFromSnapshotSourceInProgress: Cloning from a VolumeSnapshot source is in progress.
    • Paused: A multi-stage import is waiting for a new checkpoint.
    • Succeeded: The operation completed successfully.
    • Failed: The operation failed.
    • Unknown: Status is unknown.

  7. Using populators with DataVolumes

    main

    CDI automatically uses the new populator method for DataVolumes if the target storage class uses a CSI provisioner.

    Key differences when using DataVolumes with populators:

    • The created PVC will only transition to Bound once the population process is complete.
    • For storage classes using WaitForFirstConsumer (WFFC), the DataVolume status will show PendingPopulation instead of WaitForFirstConsumer.
    • DataVolumes and their created PVCs are marked with the usePopulator annotation to indicate the method used.

  8. How the Containerized Data Importer (CDI) workflow works

    main

    CDI automates the process of populating Persistent Volume Claims (PVCs) with VM disk images from remote sources. The workflow follows these steps:

    1. Initialization: An admin deploys the CDI Deployment to a target namespace (the "golden" namespace). This launches the CDI Controller.
    2. Credential Setup (Optional): If the source requires authentication, the admin creates a Kubernetes Secret in the "golden" namespace containing the necessary credentials.
    3. Triggering Import: The admin creates a "Golden" PVC in the "golden" namespace. This PVC must include specific annotations that define the source URI and the secret name.
    4. Provisioning: The Kubernetes Dynamic Provisioner (via the specified storageClass) creates a Persistent Volume (PV) to back the PVC.
    5. Execution: The CDI Controller detects the annotated PVC and launches an ephemeral Data Import Pod. This pod mounts the volume, retrieves the secret, and streams the file from the remote endpoint to the volume.
    6. Finalization: The importer performs necessary conversions (unarchiving, decompression, or qcow2-to-raw) and saves the file as disk.img. Once the copy is complete, the pod terminates.

  9. How CSI Volume Cloning works

    main

    The cloning workflow follows these steps:

    1. A DataVolume is created specifying a PVC as the source.
    2. The system checks if CSI Volume Cloning is possible based on the prerequisites (driver support, cloneStrategy=csi-clone, matching VolumeMode, etc.).
    3. If possible: The system creates the target PVC and sets the claim reference of the PV to point to the new target PVC.
    4. If not possible: The system attempts to perform Host Assisted Cloning instead.

  10. Automate OS image import and updates with DataImportCron

    main

    CDI provides the DataImportCron resource to automate the importing and periodic updating of OS images based on a schedule.

    How it works:

    1. Initial Import: On the first scheduled run, the controller imports the source image.
    2. Polling & Updates: On subsequent scheduled polls, if the source image's digest (sha256) has changed, the controller imports the new version into a new source in the DataImportCron namespace.
    3. DataSource Update: The controller automatically updates the managed DataSource to point to the newly created source.
    4. Garbage Collection: By default, garbageCollect: Outdated is enabled. The controller keeps a specified number of recent imports (defined by importsToKeep, default is 3) and deletes older ones to save space.

    To use the latest version of an automated image, create a DataVolume using a sourceRef pointing to the DataSource managed by the DataImportCron instead of defining a static source.

    apiVersion: cdi.kubevirt.io/v1beta1
kind: DataImportCron
metadata:
  name: fedora-image-import-cron
  namespace: golden-images
spec:
  template:
    spec:
      source:
        registry:
          url: "docker://quay.io/kubevirt/fedora-cloud-registry-disk-demo:latest"
          pullMethod: node
          certConfigMap: some-certs
      storage:
        resources:
          requests:
            storage: 5Gi
        storageClassName: hostpath-provisioner
  schedule: "30 1 * * 1"
  garbageCollect: Outdated
  importsToKeep: 2
  managedDataSource: fedora

  11. Understand Storage Profiles in CDI

    main

    A StorageProfile is a CDI resource that provides recommended parameters for PersistentVolumeClaims (PVCs). It allows users to simplify DataVolume (DV) definitions by omitting certain properties, as CDI will automatically apply defaults from the corresponding StorageProfile based on the storageClassName used.

    Key benefits include:

    • Simplified DataVolumes: You can omit accessModes and volumeMode in your DV spec if they are defined in the StorageProfile.
    • Automatic PVC Completion: For PVCs created independently (without a DV), the CDI PVC mutating webhook can auto-complete missing fields using StorageProfile defaults.
    • Automatic Creation: CDI automatically creates one StorageProfile for every StorageClass existing in the cluster.

    If a storage provisioner is not well-known to CDI, the resulting StorageProfile will have an empty claimPropertySets section, meaning defaults cannot be applied automatically.

  12. How PVC Mutating Webhook Rendering works

    main

    PVC Mutating Webhook Rendering is an optional CDI feature that allows users to benefit from CDI's PVC rendering logic (such as applying StorageProfiles, CDIConfig filesystemOverhead, and optimal accessMode selection) without needing to use a DataVolume object.

    When a PVC is labeled for rendering, the webhook intercepts the creation request and automatically completes missing spec fields (like volumeMode, accessMode, and storage) based on the optimal values for the specified StorageClass.

    Key behaviors:

    • Safety: The webhook only intercepts PVCs that are explicitly labeled. Unlabeled PVCs are unaffected, ensuring cluster stability even if the CDI API server is unavailable.
    • Failure Mode: If a PVC is labeled but the CDI API server is down, the request and PVC creation will fail because the webhook cannot be reached.
    • Complementary Use: This feature is designed to work alongside CDI volume populators, providing the benefits of DataVolume rendering without the architectural constraints of DataVolumes (e.g., in backup/restore or disaster recovery scenarios).