DOCSEARCHbyAlgolia
Submit Docs

Longhorn

repository·master·Indexed 11 days ago

https://github.com/longhorn/longhorn

A distributed, cloud-native block storage system for Kubernetes providing reliable, enterprise-grade persistent storage through synchronous replication, incremental snapshots, and secondary storage backups via NFS or S3.

Tokens
188.9K
Snippets
242
Records
710
Agent score
88%

What's inside Longhorn

  1. What is Longhorn?

    master

    Longhorn is a distributed, cloud-native block storage system designed specifically for Kubernetes. It is a CNCF Incubating Project that provides persistent volume support to Kubernetes clusters using containers and microservices.

    Core Architecture & Features:

    • Distributed Block Storage: Longhorn creates a dedicated storage controller for each block device volume and synchronously replicates volumes across multiple replicas on different nodes.
    • No Single Point of Failure: Designed for enterprise-grade reliability.
    • Snapshots & Backups: Supports incremental snapshots of block storage and backups to secondary storage (NFSv4 or S3-compatible object storage) using efficient change block detection.
    • Automation: Supports recurring snapshots/backups and automated, non-disruptive software upgrades.
    • Management: Includes an intuitive GUI dashboard for monitoring and management.

  2. Longhorn v1.8.0 Release Features and Improvements

    master

    Longhorn v1.8.0 introduced several significant features, particularly focusing on the V2 Data Engine and backup capabilities:

    V2 Data Engine Enhancements

    • Backing Images: Support for backing images in the CSI flow and visibility within the UI.
    • Data Locality: Support for data locality in v2 volumes.
    • Configuration: Ability to configure CPU cores for the v2 data engine and periodic HugePages (2Mi) configuration checks to ensure compatibility.
    • Resilience: Support for v2 volume delta replica rebuilding based on snapshot checksums (both data plane and control plane, including SPDK).
    • Talos Linux: Support for the v2 data engine on Talos Linux.

    Backup and Storage

    • Multiple Backupstores: Support for multiple backup stores, including UI support for managing them.
    • Default Backup Target: Separation of default backup target settings from general default settings.
    • Retention: Note that backupstores cannot set a retention policy.

    RWX Volumes

    • Automatic Expansion: Support for automatic online filesystem resize for RWX volumes.

    Other Improvements

    • CLI: Longhorn CLI now supports the darwin platform and includes checks for dmsetup and dmcrypt utilities.
    • Logging: Support for JSON log format configuration across Longhorn components (UI, driver) and configurable log levels for system and user-managed components via Longhorn manager settings.

  3. Understand Longhorn performance and scalability guidelines

    master

    Longhorn provides reference setups to help users estimate performance and scalability based on their specific cluster configurations. These guidelines cover two primary environment types: public cloud and on-prem.

    Testing and reports focus on two main areas:

    1. Data Plane Performance: Measuring read/write performance under varying workloads.
    2. Control Plane Performance: Measuring the speed of volume operations, including provisioning, attaching, detaching, mounting, and unmounting.

    Key parameters used to evaluate these environments include:

    • Number of workload pods
    • IO patterns
    • Number of Longhorn volumes
    • Number of nodes in the cluster
    • Size of Longhorn volumes

  4. Deploy test-ready Longhorn backup targets

    master

    The deploy/backupstores/ directory provides Kustomize templates to deploy test-ready backup targets for Longhorn. This allows you to set up backend storage for testing Longhorn backup functionality without hardcoding credentials in your Git repository.

    Supported backends:

    • Azurite: Azure Blob-compatible storage.
    • CIFS: SMB mount.
    • MinIO: S3-compatible object storage.
    • NFS: Network File System.

    All deployments use dynamic Kustomize patches generated via a script to inject credentials into the appropriate namespaces (default or longhorn-system).

  5. Use the Longhorn Commandline Interface (longhornctl)

    master

    The longhornctl CLI is a unified interface designed to streamline manual Longhorn operations, troubleshooting, and data retrieval. It replaces complex manual steps and extensive documentation lookups with consolidated commands for common tasks.

    Key capabilities include:

    • Installation/Uninstallation: Managing Longhorn extensions and preflight checks.
    • Operations: Performing volume trimming and replica exporting.
    • Troubleshooting: Running health checks and retrieving Longhorn resource information.
    • Data Retrieval: Exporting data from Longhorn volumes during failures to a specified path.

  6. New features in Longhorn v1.7.0

    master

    Longhorn v1.7.0 introduced several significant features and improvements, including:

    • v2 Data Engine Enhancements: Support for filesystem trim, live upgrades for the data plane, and online rebuilding of volume replicas.
    • Volume Management: Enhanced volume cloning via the UI, support for volume encryption (including encrypted backing image volumes), and auto-balancing of volumes between nodes and disks.
    • Backup & Data Protection: Support for periodic or on-demand full backups, a new BackupBackingImage UI, and the ability to add parameters to Backup and RecurringJob resources.
    • Storage & Networking: Support for storage networks for RWX volumes and the addition of nodeSelector as a parameter to VolumeSnapshotClass for backing images.
    • CLI & UI: A new Longhorn CLI focused on non-CR (Custom Resource) operations, and expanded UI support for volume cloning and backing image management.
    • Observability: Additional monitoring settings for the ServiceMonitor resource.

  7. Overview of the Longhorn SPDK Engine

    master

    The SPDK Engine is a high-performance second-generation engine for Longhorn that leverages the Storage Performance Development Kit (SPDK). Unlike the legacy engine, the SPDK engine architecture shifts data flow management to SPDK, where the engine and replicas are managed as SPDK components via spdk_tgt.

    Key Abstractions

    • Data Disk: Abstracted as an aio bdev + an lvstore.
    • Snapshot/Volume Head File: Represented as a logical volume (lvol) inside an lvstore.
    • Remote Replica: Exposed as an NVMe-oF subsystem with a corresponding SPDK lvol behind it.
    • Local Replica: Represented simply as an lvol.
    • Engine Backend: A SPDK RAID1 bdev consisting of multiple attached replica NVMe-oF subsystems and/or local lvols.
    • Engine Frontend: Typically an NVMe-oF initiator combined with an NVMe-oF subsystem of the RAID bdev.

  8. Understand the Longhorn SPDK Engine Architecture

    master

    The SPDK implementation replaces the standard longhorn-engine and longhorn-engine replicas with a more centralized model.

    • spdk_tgt: A single instance per node that manages all volumes on that node by taking control of entire disks.
    • longhorn-spdk-engine: An instance per volume that implements the gRPC interface used by the instance-manager. It acts as the orchestration layer to communicate with the longhorn-manager via a different data plane.
    • instance-manager: Acts as the portal to communicate with longhorn-manager using the longhorn-spdk-engine as the data plane.

    This architecture moves from many small engine instances to a single high-performance SPDK instance per node, with lightweight engine controllers per volume.

  9. Understand Longhorn Volume Data Sources

    master

    Longhorn uses specific URI formats to identify the source of data for a new volume clone. These are used in the v.Spec.DataSource field of the volume specification.

    Supported Data Source types:

    • Volume: Uses the format vol://<VOLUME-NAME>. This clones the current state of a volume.
    • Snapshot: Uses the format snap://<VOLUME-NAME>/<SNAPSHOT-NAME>. This clones a specific point-in-time snapshot of a volume.
    • Backup (Planned): A future format bk://<VOLUME-NAME>/<BACKUP-NAME> is intended for cloning from backups.

  10. How Replica Balance Scheduling works

    master

    Longhorn uses a balance-aware disk selection policy to prevent uneven storage usage across nodes and disks. Instead of the legacy heuristic of simply picking the disk with the most usable storage, the scheduler simulates the placement of a replica and selects the location that results in the most balanced distribution of usable storage.

    The scheduling follows a two-step hierarchy:

    1. Balance across nodes: The scheduler first identifies the node that would result in the most balanced total usable storage across all nodes.
    2. Balance across disks: Within the selected node, the scheduler then identifies the specific disk that results in the most balanced distribution of usable storage among the disks in that node.

    This approach improves fault tolerance and long-term stability by ensuring storage capacity is utilized more evenly.

  11. Configure Data Locality Strict Mode for Local Volumes

    master
    Longhorn v1.4.0 introduces a Strict Local Data Locality setting. This allows you to create a volume where one replica stays in a consistent location, and data transfer between the volume frontend and engine occurs through a local socket instead of the TCP stack. This is designed to improve performance and reduce resource consumption for distributed data workloads.

  12. Manage multiple backup targets and their lifecycle

    master

    Longhorn supports a default backup target and multiple "extra" backup targets.

    Deleting Backup Targets

    • Default Target: Deleting the default backup target is not allowed.
    • DR Volumes: If Disaster Recovery (DR) volumes exist, deleting any backup target is not allowed.
    • Extra Targets: When an extra backup target is deleted, Longhorn will automatically clean up the associated BackupVolume and Backup CR objects in the cluster. Note that the remote backup data in the actual storage (e.g., S3, NFS) will remain intact; only the cluster's metadata/references are removed.

    Handling URL Changes

    • Valid URL: The backup target controller synchronizes backup volumes and backups from the remote target to the cluster.
    • Empty URL: If the URL is emptied, the controller cleans up the associated backup volumes and backups in the cluster.
    • Invalid URL: If the URL is changed to an invalid format, the target is marked as unavailable in Status.Condition. Synchronization is skipped, but existing synchronized metadata is preserved.