democratic-csi

democratic-csi implements the Container Storage Interface (CSI) specification to provide storage for container orchestration systems like Kubernetes. It supports advanced CSI features including resizing, snapshots, and clones.

The project provides several driver implementations categorized by their backend and protocol:

ZFS-based Drivers (FreeNAS / TrueNAS)

• freenas-nfs: Manages ZFS datasets shared over NFS.
• freenas-iscsi: Manages ZFS zvols shared over iSCSI.
• freenas-smb: Manages ZFS datasets shared over SMB.
• freenas-nvmeof: Manages ZFS zvols shared over NVMe-oF.
• freenas-api-* (experimental, SCALE only): API-based versions of the above drivers.

ZFS-based Drivers (Generic ZoL, e.g., Ubuntu)

• zfs-generic-nfs: Works with any ZFS on Linux installation.
• zfs-generic-iscsi: Works with any ZFS on Linux installation.
• zfs-generic-smb: Works with any ZFS on Linux installation.
• zfs-generic-nvmeof: Works with any ZFS on Linux installation.

Local ZFS Drivers

• zfs-local-ephemeral-inline: Provisions node-local ZFS datasets.
• zfs-local-dataset: Provisions node-local volumes as datasets.
• zfs-local-zvol: Provisions node-local volumes as zvols.

Other Drivers

• synology-iscsi (experimental): Manages Synology volumes over iSCSI.
• objectivefs: Manages ObjectiveFS volumes.
• lustre-client: Provisions storage using a shared Lustre share/directory.
• nfs-client: Provisions storage using a shared NFS share/directory.
• smb-client: Provisions storage using a shared SMB share/directory.
• local-hostpath: Provisions node-local directories.
• node-manual: Connects to manually created SMB, NFS, Lustre, OneClient, NVMe-oF, or iSCSI volumes.
• containerd-oci-ephemeral-inline: Provisions ephemeral RW node-local storage using OCI images.
• vhd-ephemeral-inline: Provisions ephemeral RW node-local storage using VHD images.

For ZoL drivers (zfs-generic-*), ensure ssh and zfs are installed on the server and that targetcli is present. It is recommended to disable motd details for the SSH user to prevent CPU spikes.

Package Installation by Protocol

NFS

yum install -y nfs-utils
systemctl enable --now nfs-server.service

# OR for Debian/Ubuntu
apt-get install -y nfs-kernel-server
systemctl enable --now nfs-kernel-server.service

iSCSI

yum install targetcli -y
apt-get -y install targetcli-fb

SMB

apt-get install -y samba smbclient

# Create a POSIX user
groupadd -g 1001 smbroot
useradd -u 1001 -g 1001 -M -N -s /sbin/nologin smbroot
# Set password (required for client mount options)
smbpasswd -L -a smbroot

NVMe-over-Fabrics (nvmeof)

1. Load modules: nvmet, nvmet-tcp, nvmet-fc, nvmet-rdma.
2. Install nvmetcli and configshell_fb via pip or setup.py.
3. Configure a systemd unit (nvmet.service) to manage the service.
4. Manually create ports using nvmetcli (e.g., TCP port 4420 on 0.0.0.0).

# Example: NFS setup on RHEL
yum install -y nfs-utils
systemctl enable --now nfs-server.service

To enable CHAP Authentication, you must provide a Kubernetes Secret referenced by the StorageClass.

Configuration Steps

1. Create a Secret containing the lunTemplate, targetTemplate, and lunSnapshotTemplate.
2. Define Credentials in the targetTemplate section of the secret:
   • user: Client username
   • password: Client password
   • mutualUser: (Optional) Server username for mutual CHAP
   • mutualPassword: (Optional) Server password for mutual CHAP
3. Reference the Secret in your StorageClass using the following keys:
   • csi.storage.k8s.io/provisioner-secret-name
   • csi.storage.k8s.io/provisioner-secret-namespace

Usage Patterns

• Global Credentials: Use a static secret name for all volumes in the StorageClass.
• Per-PVC Credentials: Use substitution syntax (e.g., ${pvc.name}-chap-secret) to allow different credentials per volume.

Important Notes

• CHAP is only enabled if the secret contains both a username and a password. If a password is missing, the volume is still created but without CHAP.
• You cannot change the password or toggle CHAP after the volume has been created.
• If the referenced secret is missing, the volume creation will fail.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: synology-iscsi-chap
parameters:
  fsType: ext4
  lunTemplate: |
    type: BLUN
    description: iSCSI volumes with CHAP Authentication
secrets:
  # Use this to configure a single set of credentials for all volumes of this StorageClass
  csi.storage.k8s.io/provisioner-secret-name: chap-secret
  csi.storage.k8s.io/provisioner-secret-namespace: default
  # Use substitutions to use different credentials for volumes based on the PVC
  csi.storage.k8s.io/provisioner-secret-name: "${pvc.name}-chap-secret"
  csi.storage.k8s.io/provisioner-secret-namespace: "${pvc.namespace}"
---
# Use a secret like this to supply CHAP credentials.
apiVersion: v1
kind: Secret
metadata:
  name: chap-secret
stringData:
  lunTemplate: |
    ...
  targetTemplate: |
    # Client Credentials
    user: client
    password: MySecretPassword
    # Mutual CHAP Credentials. If these are specified mutual CHAP will be enabled.
    mutualUser: server
    mutualPassword: MyOtherPassword
  lunSnapshotTemplate: |
    ...

After creating a volume externally, you must register it with Nomad so it can be used in jobs. While the Nomad CLI can do this, using the Terraform nomad_volume resource is recommended.

When registering, you must map the metadata returned by the csc command (such as node_attach_driver, provisioner_driver, server, and share) into the context block of the Terraform resource.

provider "nomad" {
  address = "<nomad address>"
}

resource "nomad_volume" "<volume name>" {
  type                  = "csi"
  plugin_id             = "truenas"
  volume_id             = "<volume name>"
  name                  = "<volume name>"
  external_id           = "<volume name>"
  access_mode           = "single-node-writer"
  attachment_mode       = "file-system"
  deregister_on_destroy = true

  mount_options = {
    fs_type = "nfs"
  }

  context = {
    node_attach_driver = "nfs"
    provisioner_driver = "freenas-nfs"
    server             = "<server>"
    share              = "<share>"
  }
}