DOCSEARCHbyAlgolia
Submit Docs

OpenEBS Documentation

repository·develop·Indexed 27 days ago

https://github.com/openebs/openebs

OpenEBS is an open-source, container-native storage solution for Kubernetes providing persistent storage through dynamic provisioning and containerized storage controllers. It features a multi-plane architecture consisting of a Data Plane (including Local PV drivers like ZFS, LVM, Hostpath, and Rawfile, as well as the replicated Mayastor engine), a Control Plane for storage management via CSI, and a Storage Device Management Plane. The project includes a unified Helm chart for installation and the kubectl-openebs plugin for CLI management.

Tokens
38.9K
Snippets
73
Records
209
Agent score
92%

What's inside OpenEBS

  1. Overview of OpenEBS Architecture

    develop

    OpenEBS is a container-native, horizontally scalable Container Attached Storage (CAS) solution for Kubernetes. It is composed of three primary architectural planes:

    1. Data Plane (Data Engines): Containers that interface with underlying storage (host filesystems, SSDs, NVMe, etc.) and provide volume capabilities like high availability, snapshots, and clones. Examples include cStor, Jiva, and Local PVs.
    2. Control Plane (Storage Management): Manages the interface between Kubernetes (via CSI) and the Data Engines. It provides APIs for volume management, metrics collection (e.g., for Prometheus), and status insights.
    3. Storage Device Management Plane: An inventory management layer that discovers and tracks storage devices (rotational drives, SSDs, NVMe) using Kubernetes Custom Resources and device claims.

  2. Overview of OpenEBS Storage Approaches

    develop

    OpenEBS provides two primary storage approaches for Kubernetes workloads: Local Storage and Replicated Storage. Choosing between them depends on whether your application manages its own replication or requires storage-level high availability.

    Local Storage

    • Data Availability: Limited to the node where the volume is provisioned. Node failure leads to data unavailability.
    • Use Cases: Ideal for distributed databases that manage their own replication (e.g., MongoDB, Cassandra).
    • Performance: Near-disk performance with minimal overhead.
    • Features: Supports snapshots and cloning when using advanced filesystems like LVM or ZFS. Backup/Restore is supported via Velero using Restic.

    Replicated Storage

    • Data Availability: Synchronously replicates data across multiple nodes, ensuring high availability and durability.
    • Use Cases: Suitable for stateful workloads requiring storage-level replication (e.g., Percona, Standalone DBs, GitLab).
    • Performance: High performance leveraging NVMe-oF semantics for low-latency access.
    • Features: Supports enterprise capabilities like snapshots, clones, and high availability. Backup/Restore is supported via Velero.

  3. Understand DiskPool failure modes and handling

    develop

    OpenEBS Mayastor handles various disk failure scenarios to ensure data integrity and system stability. The handling depends on the specific symptom:

    Failure ModeSymptomHandling
    Physical hot-removal / full device lossFile descriptor invalid; I/O returns ENODEV/EIObdev auto-unregistration → pool unload → Offline
    Bad sectors (localised corruption)Reads at specific LBAs fail with EIOError counter increment; alert escalation
    Intermittent read/write errorsSector sometimes succeeds; long latency spikesError counter + stall transition tracking
    Controller / firmware glitchRandom EIO across different offsetsError counter
    Power instability (SSDs)Incomplete writes during erase/program cycleError counter; InvalidSuperBlock on next import

    Note: This system focuses on error counting and alerting. SMART-based predictive analysis is handled separately.

  4. Volume Group Snapshot feature in OpenEBS ZFS

    develop
    OpenEBS ZFS is proposing a Volume Group Snapshot feature to allow users to create consistent snapshots of multiple volumes in a single operation. This is specifically designed for stateful applications with interdependent datasets (like distributed databases) that require simultaneous snapshots to ensure data consistency across all related volumes.

  5. Understand DiskPool failure handling in Mayastor

    develop

    OpenEBS Mayastor implements a multi-layered response to DiskPool backing-disk failures, hot-removals, persistent I/O errors, and I/O stalls. This system prevents the control-plane from entering futile retry loops and provides actionable status information to users.

    Key capabilities include:

    • Automatic bdev eviction: Unregisters the SPDK block device (bdev) immediately upon hot-removal.
    • Pool Probing: Uses a PoolProbe gRPC API to verify device presence and I/O health before and after import attempts.
    • Structured Status: Replaces generic Unknown statuses with specific reason codes like DiskNotFound, DiskReadIoError, ForeignPoolName, and InvalidSuperBlock.
    • Health States: Distinguishes between Offline (device gone), Faulted (I/O errors), and Unknown (node unreachable).
    • I/O Stall Detection: Detects when the I/O queue is frozen beyond a deadline, transitions the pool to a Critical alert state, and attempts a bdev reset.
    • Alert Hierarchy: Uses a four-level classification (Healthy, Attention, Warning, Critical) based on error counts and stall transitions to inform the control-plane scheduler.

  6. Understand Offline Volume Rebuild in Mayastor

    develop

    In Mayastor, volumes that are unpublished (not currently mounted/active via a nexus) do not automatically trigger replica rebuilds when they become Degraded. This is because the existing hot-spare logic requires an active nexus to function.

    The Offline Volume Rebuild mechanism addresses this by detecting degraded unpublished volumes and automatically triggering a rebuild after a configurable threshold. The control plane achieves this by:

    1. Detecting a Degraded state via NexusInfo health signals.
    2. Temporarily creating a non-shared nexus.
    3. Allowing the existing rebuild engine to restore replicas.
    4. Tearing down the temporary nexus once the rebuild is complete.

    This is particularly useful for CDI golden images, cold backup volumes, and volumes attached to scaled-down workloads that may experience replica loss during node maintenance or disk decommissioning.

  7. Understand ThinPool based Scheduling in LVM LocalPV

    develop

    OpenEBS LVM LocalPV uses a ThinPool (a Logical Volume created on a Volume Group) to host thin LVs. This enhancement ensures the scheduler correctly accounts for both the Volume Group (VG) capacity and the specific ThinPool free space when making scheduling decisions for thin PVCs.

    Key scheduling behaviors:

    • Thin PVCs: In the SpaceWeighted algorithm, the scheduler includes thinpool free space when searching for a Volume Group with maximum free space on a node.
    • Thick PVCs: The scheduler verifies that the Volume Group has sufficient physical space to host the Logical Volume (LV) across all algorithms. If the VG lacks sufficient space, scheduling fails.
    • Algorithm Updates: CapacityWeighted and VolumeWeighted algorithms have been updated to use nodelist instead of volumelist to ensure all Volume Groups are considered, even those without existing volumes.

  8. Volume Group Snapshot Feature in OpenEBS Mayastor

    develop

    OpenEBS Mayastor is introducing a Volume Group Snapshot feature to allow users to create consistent snapshots of multiple volumes in a single operation. This is specifically designed for stateful applications with interdependent datasets (like distributed databases) that require simultaneous snapshots to ensure data consistency across all related volumes.

    Key capabilities include:

    • Creating write-consistent snapshots across an entire volume group.
    • Deleting volume group snapshots when they are no longer needed.
    • Compatibility with existing snapshot and backup workflows.

  9. Volume Group Snapshot Feature in OpenEBS LVM

    develop

    OpenEBS LVM is introducing a Volume Group Snapshot feature to allow users to create consistent snapshots of multiple volumes in a single operation. This is specifically designed for stateful applications with interdependent datasets (e.g., distributed databases or clustered services) that require simultaneous snapshots to ensure data consistency across all related volumes.

    Key Capabilities:

    • Create consistent snapshots across multiple application volumes simultaneously.
    • Delete volume group snapshots when they are no longer needed.
    • Compatibility with existing snapshot and backup workflows.

    Note: As of the current proposal, the Snapshot Restore workflow is not included in this feature set and may require manual LVM-specific steps.

  10. Enable TLS in Mayastor

    develop
    Mayastor supports TLS communication between its components (rest-api, core-agent, and io-engine) to ensure data integrity and confidentiality. TLS can be enabled via a boolean switch in the values.yaml file. When enabled, Mayastor utilizes cert-manager to automate certificate creation and management. Components are designed to fallback to HTTP if TLS is disabled, and use HTTPS when enabled.

  11. Install OpenEBS Helm chart

    develop

    Install the unified OpenEBS Helm chart with default values. This installs LocalPV Hostpath, LVM, ZFS, and Mayastor components in the openebs namespace.

    Note: Ensure you have met the specific prerequisites for the storage engines you intend to use (Hostpath, LVM, ZFS, or Mayastor) before installation.

    helm install openebs --namespace openebs openebs/openebs --create-namespace

  12. Identify required container images for OpenEBS Helm charts

    develop

    OpenEBS Helm charts include an images annotation in their Chart.yaml (or doc.yaml) file. This annotation provides a single source of truth for all container images required by the chart, which is particularly useful for air-gapped environments where images must be pre-downloaded.

    To discover the images used by a chart via the command line, you can template the Helm chart and use a regular expression to extract the image strings. It is recommended to enable all features during this process to ensure a comprehensive list.

    helm template . --set "$ENABLE_ALL_FEATURES" | grep -Po "^[ \t]*image: \K(.*:.*)$" | tr -d \"