DOCSEARCHbyAlgolia
Submit Docs

Proxmox CSI Plugin

repository·main·Indexed 21 days ago

https://github.com/sergelogvinov/proxmox-csi-plugin

A Kubernetes CSI plugin for dynamic volume provisioning and management on Proxmox hypervisors. It supports storage types including Directory, LVM, LVM-thin, ZFS, NFS, and Ceph. Key features include topology-aware scheduling, volume expansion, LUKS encryption, snapshots, and ZFS replication. The plugin maps Proxmox clusters to Kubernetes regions and Proxmox nodes to Kubernetes zones.

Tokens
19.4K
Snippets
60
Records
74
Agent score
73%

What's inside proxmox-csi-plugin

  1. Overview of Proxmox CSI Plugin capabilities

    main

    The Proxmox CSI Plugin enables Kubernetes to manage persistent storage on Proxmox hypervisors. It allows pods to access data as attached block devices, providing greater flexibility for migrating pods between VMs within a Proxmox node.

    Key Features:

    • Dynamic Provisioning: Volumes are created automatically when PersistentVolumeClaim objects are created.
    • Topology Awareness: Supports scheduling Pods to specific Nodes where the disk volume pool exists.
    • Volume Expansion: Volumes can be resized by editing PersistentVolumeClaim objects.
    • Volume Metrics: Exports usage statistics as Prometheus metrics via kubelet.
    • Storage Capacity: Exposes Proxmox storage capacity to the Kubernetes controller.
    • Encrypted Volumes: Supports LUKS encryption.
    • Volume Bandwidth: Supports setting maximum read/write limits.
    • Volume Snapshots: Supports creating and restoring volume snapshots.
    • Volume Migration: Supports offline migration of PVs to another Proxmox node (region) via the pvecsictl tool.
    • Zone Replication: Supports ZFS replication to another Proxmox node (zone).

  3. Understand Pod migration behavior in Proxmox clusters

    main

    The Proxmox CSI plugin manages storage for Proxmox clusters using local storage (e.g., lvm, lvm-thin, zfs, xfs, ext4). Understanding how Pods and Persistent Volumes (PVs) migrate depends on the relationship between Proxmox Nodes and Kubernetes Zones:

    • Proxmox Cluster: A collection of nodes. Each cluster has a predefined region name configured in cloud-config (referenced as clusters[].region).
    • Proxmox Node (Kubernetes zone): In Kubernetes, a Proxmox Node is represented as a zone. The zone name corresponds to the Proxmox node name.

    Migration Rules

    1. Within a Zone (Automatic): Pods can migrate between Kubernetes nodes that reside on the same physical Proxmox node (zone). When this happens, the CSI Plugin automatically moves the PV to the new location.
    2. Across Zones (Manual): Pods with PVCs cannot automatically migrate across different Proxmox nodes (zones). To move a Pod across zones, you must manually migrate the PV using the pvecsictl tool.

  4. How Proxmox regions and zones map to Kubernetes topology

    main

    The plugin uses a hierarchical model to manage storage across multiple Proxmox clusters and nodes:

    • Region: Corresponds to a Proxmox Cluster. Each cluster has a predefined region name in its cloud-config. In Kubernetes, this is represented by the topology.kubernetes.io/region label.
    • Zone: Corresponds to a Proxmox Node. In Kubernetes, this is represented by the topology.kubernetes.io/zone label.

    Migration Behavior:

    • Local Storage (e.g., LVM, ZFS, XFS, EXT4): Pods can automatically migrate between Kubernetes nodes on the same physical Proxmox node (zone). However, they cannot automatically migrate across different zones (Proxmox nodes). Manual migration is required using pvecsictl.
    • Shared Disk (e.g., Ceph, NFS): Pods can automatically migrate across zones within the same region.

  5. Change an encrypted disk secret key

    main

    Secret keys for encrypted disks cannot be changed via the Kubernetes API. You must use cryptsetup via the proxmox-csi-plugin pod running on the same node as the pod using the PVC.

    Steps:

    1. Identify the target disk (e.g., /dev/sdb) and the plugin pod name.
    2. Verify the current disk state and passphrase using luksDump and luksOpen --test-passphrase.
    3. Add the new passphrase using luksAddKey.
    4. Verify the new passphrase works.
    5. Remove the old passphrase using luksRemoveKey.
    6. Update the passphrase in the corresponding Kubernetes Secret resource.
    # Check the disk
kubectl -n csi-proxmox exec -ti <plugin-pod-name> -- /sbin/cryptsetup luksDump /dev/sdb

# Check the passphrase
kubectl -n csi-proxmox exec -ti <plugin-pod-name> -- /sbin/cryptsetup luksOpen --test-passphrase -v /dev/sdb

# Add the new passphrase
kubectl -n csi-proxmox exec -ti <plugin-pod-name> -- /sbin/cryptsetup luksAddKey /dev/sdb

# Verify the new passphrase
kubectl -n csi-proxmox exec -ti <plugin-pod-name> -- /sbin/cryptsetup luksOpen --test-passphrase -v /dev/sdb

# Remove the old passphrase
kubectl -n csi-proxmox exec -ti <plugin-pod-name> -- /sbin/cryptsetup luksRemoveKey /dev/sdb

  6. Configure disk encryption secrets

    main

    To enable disk encryption, you must create a Kubernetes Secret containing an encryption-passphrase. The StorageClass then references this secret via node-stage-secret-name and node-stage-secret-namespace.

    Important: If you intend to use volume expansion, the node-expand-secret-name and node-expand-secret-namespace in your StorageClass must be identical to the stage secret parameters.

    apiVersion: v1
data:
  encryption-passphrase: base64-encode
kind: Secret
metadata:
  name: proxmox-csi-secret
  namespace: kube-system

  7. Configure RBAC permissions for pvecsictl

    main

    To function correctly, pvecsictl requires a ClusterRole with the following permissions to manage pods, CSI nodes, persistent volumes, persistent volume claims, and nodes (for cordoning/uncordoning).

    apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: pvecsictl
rules:
  # Get list of pods with PVCs
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list", "watch", "delete"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["csinodes"]
    verbs: ["get", "list"]
  # Create and delete PV/PVC
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "patch", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "create", "patch", "delete"]
  # Node cordoning/uncordoning
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "watch", "patch"]

  8. Use Terraform with Proxmox CSI

    main

    When managing VMs with Terraform, you must set ignore_changes for the disk attribute. Otherwise, Terraform will attempt to remove dynamic PV/PVCs during every apply cycle.

    # bpg/proxmox plugin

resource "proxmox_virtual_environment_vm" "vm" {

  lifecycle {
    ignore_changes = [
      disk,
    ]
  }
}

  10. Run disk benchmarks on the host or in a Pod

    main

    Execute the onedisk.sh script to measure I/O performance. You can run this directly on the Proxmox host machine or within a Kubernetes Pod using a Persistent Volume Claim (PVC) to compare raw host performance against virtualized storage performance.

    # Example: Run 4 jobs on a file with 4k block size
sh onedisk.sh 4 /root/file 4k

# Example: Run 4 jobs on a file with 16k block size
sh onedisk.sh 4 /root/test 16k

# Example: Run benchmarks on a mounted PVC in a Pod
sh onedisk.sh 4 /mnt/test 4k

  11. Verify CSIDriver and storage capacity

    main

    You can monitor the health and capacity of your Proxmox storage through Kubernetes resources.

    1. Check CSIDriver existence: Ensure csi.proxmox.sinextra.dev is registered with STORAGECAPACITY set to true.
    2. Check Proxmox pool capacity: Use the csistoragecapacities resource to view available capacity per storage class and zone. Ensure the AVAIL value is non-zero.
    3. Check node CSI drivers: Verify that the driver is correctly registered on your worker nodes and check the supported topologyKeys (e.g., topology.kubernetes.io/region and topology.kubernetes.io/zone).
    # Check CSIDriver
kubectl get CSIDriver

# Check Proxmox pool capacity
kubectl get csistoragecapacities -ocustom-columns=CLASS:.storageClassName,AVAIL:.capacity,ZONE:.nodeTopology.matchLabels -A

# Check node CSI drivers on a specific node
kubectl get CSINode <node-name> -oyaml

  12. Configure Kubernetes Node Annotations and Labels for Proxmox CSI

    main

    You can provide additional information to the Proxmox CSI Driver by applying specific annotations and labels to your Kubernetes Nodes. These settings help the driver identify the Proxmox VM ID, manage topology, and limit volume attachments.

    Key Configuration Options

    Annotations

    • proxmox.sinextra.dev/instance-id: (Optional) Set this to the Proxmox Virtual Machine ID to help identify the node.

    Labels

    • topology.proxmox.sinextra.dev/region: (Optional) Override the Proxmox-specific region label.
    • topology.proxmox.sinextra.dev/zone: (Optional) Override the Proxmox-specific zone label.
    • topology.kubernetes.io/region: (Optional) Set the standard Kubernetes region label. The Kubernetes scheduler uses this.
    • topology.kubernetes.io/zone: (Optional) Set the standard Kubernetes zone label. The Kubernetes scheduler uses this.
    • csi.proxmox.sinextra.dev/max-volume-attachments: (Optional) Define the maximum number of volumes that can be attached to this node. The default is 24.
      • Note: There is a hard limit of 30 virtio iscsi volumes total (including root disks) per VM in QEMU/Proxmox.

    Spec Field

    • spec.providerID: (Optional, recommended) Use the format proxmox://<cluster-name>/<VM-ID> to help identify the node in the Proxmox cluster.
    apiVersion: v1
kind: Node
metadata:
  annotations:
    proxmox.sinextra.dev/instance-id: "VM-ID"
  labels:
    topology.proxmox.sinextra.dev/region: cluster-1
    topology.proxmox.sinextra.dev/zone: pve-node-1
    topology.kubernetes.io/region: cluster-1
    topology.kubernetes.io/zone: pve-node-1
    csi.proxmox.sinextra.dev/max-volume-attachments: "24"
spec:
  providerID: proxmox://cluster-1/VM-ID