OpenEBS Dynamic LocalPV Provisioner

OpenEBS Dynamic LocalPV Provisioner is a Kubernetes component designed to automate the dynamic provisioning of local persistent volumes. It converts local node storage (such as hostPath directories) into Persistent Volumes (PVs) that can be consumed via Persistent Volume Claims (PVCs).

Key Capabilities:

• Dynamic Provisioning: Automatically creates hostpath volumes on demand, eliminating the need for manual static provisioning.
• Node Affinity: Automatically assigns node affinity metadata to PVs, ensuring that application pods are scheduled on the specific node where the local storage resides.
• High Performance: Optimized for low-latency, high-performance stateful applications (e.g., replicated databases) that require direct access to local disk.
• Customizable: Supports configuration via StorageClasses, including features like quota enforcement.

The App Pod Failure chaos experiment is designed to test the recovery workflow of an application by disrupting its access to storage.

Procedure:

1. The experiment kills the application container (using the appropriate utility based on the Container Runtime Interface/CRI in use, such as pod_failure_by_sigkill.yaml).
2. The experiment observes the behavior of the corresponding OpenEBS Persistent Volume (PV) and the application consuming that volume.
3. The experiment verifies if the container is successfully rescheduled and if the data remains intact.

Success Criteria (Exit-Criteria):

• Application services are accessible and pods are healthy.
• Data written prior to the chaos is successfully retrieved/read.
• Database consistency is maintained (verified via integrity check utils).
• Storage target pods remain healthy.

The e2e-tests repository is organized as follows:

• apps/: Test scripts for deploying/deprovisioning applications or StatefulSets.
• chaoslib/: Collection of chaos utilities used within test scripts.
• experiments/: The core test scenarios (functional, chaos, and infra-chaos). Each experiment directory typically contains:
  • README.md: Scenario description.
  • run_e2e_test.yml: Kubernetes job specification.
  • test_vars.yml: Variables for the Ansible playbook.
  • test.yml: The actual Ansible playbook logic.
• hack/: RBAC, CRD, and test-result update manifests.
• utils/: General-purpose playbooks for Kubernetes object lifecycle management.

The provisioner manages the lifecycle of local storage through a coordinated flow between the Provisioner, a Helper Pod, and the Kubernetes control plane:

1. Request: An Application Pod requests storage via a Persistent Volume Claim (PVC).
2. Detection: The LocalPV Provisioner watches for new PVC requests.
3. Directory Creation: The Provisioner launches a Helper Pod on the target node. This Helper Pod is responsible for creating and/or cleaning up the specific directory on the host path.
4. PV Creation: The Provisioner creates a Persistent Volume (PV) that points to the hostPath directory.
5. Binding: The PV is bound to the PVC.
6. Mounting: The Kubelet on the worker node mounts the host path into the Application Pod.

This architecture ensures that the storage directory exists on the correct node before the volume is bound and used.

To install the OpenEBS Dynamic LocalPV Provisioner using Helm, use the helm install command. It is recommended to install it into a dedicated namespace (e.g., openebs).

By default, this installation includes the creation of the openebs-hostpath StorageClass.

helm install openebs-localpv openebs-localpv/localpv-provisioner --namespace openebs --create-namespace

To install the OpenEBS Dynamic LocalPV Provisioner using Helm, you must first add the official OpenEBS LocalPV repository to your local Helm configuration and update your local chart cache.

helm repo add openebs-localpv https://openebs.github.io/dynamic-localpv-provisioner
helm repo update

To use OpenEBS LocalPV Hostpath on a Talos cluster, you must bind-mount the hostpath directories to the kubelet containers. This is achieved by editing the KubeletConfig section of your cluster machineconfig (or using config patches) to include extraMounts.

If you use a custom hostpath directory instead of the default, ensure you bind-mount the absolute path of that custom directory.

kubelet:
    extraMounts:
          #Default Hostpath directory
        - destination: /var/openebs/local
          type: bind
          source: /var/openebs/local
          options:
            - rbind
            - rshared
            - rw

You can install the provisioner by applying the openebs-operator-lite.yaml and openebs-lite-sc.yaml manifests. Because MicroK8s uses a specific directory structure, you must use sed to redirect the default /var/openebs paths to /var/snap/microk8s/common/var/openebs during the application process.

# Apply openebs-operator-lite.yaml with path redirection
curl -fSsL https://openebs.github.io/charts/openebs-operator-lite.yaml | sed 's|\(/var/openebs\)|/var/snap/microk8s/common\1|g' | kubectl apply -f -

# Apply openebs-lite-sc.yaml with path redirection
curl -fSsL https://openebs.github.io/charts/openebs-lite-sc.yaml | sed 's|\(/var/openebs\)|/var/snap/microk8s/common\1|g' | kubectl apply -f -

Experiments are declarative Kubernetes jobs. To run a specific test scenario (e.g., a chaos experiment like app_pod_failure):

1. Locate the experiment: Find the desired test in the experiments/<type> directory.
2. Configure the job: Open the run_e2e_test.yml file within the experiment directory. Update the environment variables (ENVs) to match your target application's namespace, labels, and other test-specific requirements.
3. Execute: Create the Kubernetes job using the run_e2e_test.yml manifest.

kubectl create -f experiments/chaos/app_pod_failure/run_e2e_test.yml

Install the OpenEBS LocalPV Hostpath Provisioner using the official operator-lite manifest. After installation, verify that the provisioner pod is running in the openebs namespace.

kubectl apply -f https://openebs.github.io/charts/openebs-operator-lite.yaml

# Verify installation
kubectl get pods -n openebs

If your XFS filesystem is the root filesystem (/), you must modify the kernel boot options via GRUB.

1. Edit GRUB configuration: Open /etc/default/grub and find the GRUB_CMDLINE_LINUX variable. Append rootflags=pquota to the end of the string (or add pquota to the existing rootflags list).

   GRUB_CMDLINE_LINUX="... rootflags=pquota"

2. Backup and Regenerate GRUB: Locate your grub.cfg (e.g., /boot/grub2/grub.cfg) and perform the following:

   sudo cp /boot/grub2/grub.cfg /boot/grub2/grub.cfg.backup
   sudo grub2-mkconfig -o /boot/grub2/grub.cfg

3. Reboot:

   sudo reboot

4. **Verify:**
   After rebooting, confirm the option is active:
   ```bash
sudo mount | grep "^/dev/nvme0n1p1"