Vitastor Distributed Software Defined Storage

Vitastor is a reliable clustered block storage system designed without single points of failure. Key server-side capabilities include:

• High Performance: Supports NVMe atomic writes to minimize Write Amplification (targeting WA=1).
• Fault Tolerance Schemes: Supports replication, XOR n+1 (parity disk), and Reed-Solomon erasure coding (via jerasure or ISA-L) with configurable data and parity counts.
• Data Distribution: Automatic distribution across OSDs using mathematical optimization to ensure uniformity and minimize data movement. Supports:
  • Multiple pools with different redundancy schemes.
  • Distribution trees with device class selection (e.g., only SSD, only HDD).
  • Customizable failure domains (disk, server, rack, etc.).
• Storage Management: Includes degraded block recovery, rebalancing (moving data between OSDs), snapshots, and copy-on-write clones.
• Network & Optimization: Supports RDMA/RoCEv2 (via libibverbs or RDMA-CM), local reads for multi-datacenter optimization, and "lazy" fsync support.
• Integrity & Security: Background integrity scrubbing, checksums, and AES-256-XTS end-to-end encryption with Vault/OpenBao support for key management.
• Observability: Statistics collection in etcd (I/O, space usage per inode) and a built-in Prometheus exporter.

Vitastor configuration is organized into several functional areas. Depending on your deployment needs, you may need to configure:

• Parameters (Key-Value): General settings categorized by scope.
• Pool Settings: Configuration for storage pools.
• OSD Tree Settings: Placement tree configurations.
• Individual OSD Settings: Specific configurations for OSDs.
• Inode Settings: Metadata settings for images (e.g., name, size, and parent image references).

Parameter Categories:

• Common: General system settings.
• Network: Networking and connectivity.
• Client Code: Settings for client-side logic.
• Global Disk Parameters: Cluster-wide disk layout settings.
• OSD Disk Parameters: Disk-specific layout settings.
• Other OSD Parameters: Miscellaneous OSD settings.
• Monitor Parameters: Settings for Vitastor monitors.
• Security Parameters: Security and access control settings.

Vitastor provides a highly-available block storage system with symmetric clustering and no single point of failure (SPOF). Key server-side capabilities include:

Data Redundancy & Reliability

• Redundancy Schemes: Supports Replication, XOR n+1, and Reed-Solomon erasure codes (using jerasure and ISA-L libraries).
• Data Integrity: Uses checksums and scrubbing (automated verification of copies).
• Recovery: Automatic recovery of degraded blocks and rebalancing (data movement between OSDs).

Performance & Optimization

• NVMe Support: Supports atomic writes to reduce write amplification (aiming for Write Amplification = 1).
• Network: Supports RDMA/RoCEv2 via libibverbs or RDMA-CM.
• I/O Management: Features write throttling for smoothing random write workloads in hybrid SSD+HDD configurations, lazy fsync support, and localized read support for cross-datacenter optimizations.
• Caching: Supports client write-back caching.

Management & Configuration

• Configuration: Managed via JSON data structures stored in etcd (parameters, pools, and images).
• Observability: Built-in Prometheus metric exporter and per-OSD/per-image I/O and space usage statistics in etcd.
• Storage Features: Supports snapshots and copy-on-write image clones.

Starting from version 3.1.0, Vitastor provides three layers of data protection. All security features are optional and disabled by default in their simplest configuration. By default, only transport-layer data checksums (proto_checksums=payload) are enabled for supporting clients (>= 3.1.0). For older clients, connections without checksums are allowed by default (force_proto_checksums is empty).

Security Layers:

1. Control Plane Protection (etcd):

   • TLS encryption for connections to etcd.
   • Client authentication via TLS (X.509) certificates.
   • Client access control to etcd data.

2. Data Plane Protection (OSD):

   • Full AES-GCM encryption for connections to OSD (faster than TLS).
   • Alternative: AES-GCM encryption for command headers only, using data checksums with a secret "salt".
   • Client authentication via TLS (X.509) certificates.
   • Client access control on the OSD side.

3. End-to-End Encryption:

   • Data is encrypted using AES-XTS on the client side; the Vitastor cluster never sees plaintext data.
   • AES-XTS keys can be stored in etcd or an external Vault/OpenBao instance.

Vitastor is licensed under the VNPL 1.1 (Vitastor Network Public License). This is a copyleft license based on GNU GPLv3.0 that includes a "Network Interaction" clause.

Unlike standard AGPL, VNPL extends copyleft requirements to Proxy Programs: micro-service modules that interact with Vitastor over a network and are specifically designed to work with it. If you use Vitastor in a way that provides its functionality to users via a Proxy Program, you must open-source those Proxy Programs under a GPL-compatible license (e.g., GPL, LGPL, MIT/BSD, or Apache 2) unless you obtain a commercial license.

Key distinction:

• Proxy Programs: Software designed specifically for Vitastor that interacts with it via API, CLI, or network (e.g., a Vitastor CSI driver). These must be open-sourced.
• Non-Proxy Programs: Software that interacts with Vitastor indirectly through generic interfaces and was not designed for Vitastor (e.g., an OS running in a VM using a Vitastor-backed block device). These do not require open-sourcing.

To determine if your software is subject to the VNPL open-source requirements, evaluate it against two criteria. A program is a Proxy Program if it meets both conditions:

1. Special Design: The program has functionality specific to Vitastor and "knows" it is working with Vitastor (e.g., it handles Vitastor-specific snapshots or features).
2. Interaction: It interacts with Vitastor directly or indirectly through any interface, including APIs, CLIs, network protocols, or wrappers.

Decision Logic Examples:

• CSI Driver: Interacts via vitastor-cli and is designed specifically for Vitastor $\rightarrow$ IS a Proxy Program (must be open-sourced).
• Operating System in a VM: Interacts via a generic block device interface (e.g., QEMU) and was not designed for Vitastor $\rightarrow$ IS NOT a Proxy Program.
• Cloud Control Panel: If the panel's functionality changes when Vitastor is replaced (e.g., specific snapshot management), it is designed for Vitastor $\rightarrow$ IS a Proxy Program. If the panel is universal and doesn't change $\rightarrow$ IS NOT a Proxy Program.