AWS EBS CSI Driver

The Amazon Elastic Block Store (EBS) CSI Driver provides a Container Storage Interface (CSI) to manage the lifecycle of Amazon EBS volumes and snapshots within Kubernetes.

Key capabilities include:

• Static Provisioning: Manually associating existing EBS volumes with a Kubernetes PersistentVolume (PV).
• Dynamic Provisioning: Automatically creating EBS volumes and PersistentVolumes (PV) from PersistentVolumeClaims (PVC) using a StorageClass for configuration.
• Mount Options: Defining how volumes are mounted via the PersistentVolume (PV) resource.
• Block Volumes: Using EBS volumes as raw block devices.
• Volume Snapshots: Managing Kubernetes snapshots to create or restore EBS volumes.
• Volume Resizing: Expanding volume capacity by updating the PersistentVolumeClaim (PVC) size.
• Volume Modification: Changing volume properties like type, IOPS, or throughput using a VolumeAttributesClass.
• Node-Local Volumes: Using a single cluster-wide PV/PVC to mount pre-attached, node-specific EBS volumes for local caching.

To avoid hitting AWS EC2 ModifyVolume API cooldowns, the driver implements request coalescing.

When a user updates a PersistentVolumeClaim (PVC) with a new size (triggering ControllerExpandVolume) and a new VolumeAttributesClassName (triggering ControllerModifyVolume) simultaneously, the driver waits up to two seconds to merge these into a single EC2 ModifyVolume call.

The workflow is as follows:

1. The csi-resizer sends both RPCs to the ebs-plugin.
2. The plugin merges the requests and calls EC2 CreateTags (if needed) and EC2 ModifyVolume.
3. The plugin polls EC2 DescribeVolumeModifications until the state is optimizing or completed.
4. If the volume is a Block Device, the expansion is considered successful.
5. If the volume is a Filesystem, the driver marks the PVC as FSResizeRequired, triggering the Kubelet to call NodeExpandVolume for an online filesystem resize.

The ebs-plugin is designed around several critical operational requirements to ensure reliability in distributed environments:

• Idempotency: All CSI calls (e.g., CreateVolume, ControllerPublish, DeleteVolume) must be idempotent. The plugin must check the current state of AWS before acting. For example, CreateVolume should return the existing volume if it was already provisioned, and ControllerPublish should return success if the volume is already attached to the requested node.
• Timeout Handling: Since gRPC calls can timeout while the underlying AWS operation continues, the driver must be able to handle retries from Kubernetes sidecars. It must check if a previously timed-out operation (like an attachment) actually succeeded before attempting it again.
• Statelessness/Restarts: The driver is designed to be stateless. Upon restart or crash, it recovers its state by observing the actual status of AWS resources (e.g., describing instances and volumes).
• Deterministic Device Naming: Because AWS requires the client to assign device names and imposes restrictions on them, the driver assigns device names in a deterministic order and maintains a cache of attempted names that are likely unusable for a specific instance type.

The driver operates via two main service types: the Identity Service and the Controller Service, which handle high-level volume management, and the Node Service, which handles local device operations on Kubernetes nodes.

Identity Service

Used by Kubernetes to verify driver health and capabilities:

• GetPluginInfo: Returns driver name (ebs.csi.aws.com) and version.
• GetPluginCapabilities: Returns supported capabilities like CONTROLLER_SERVICE.
• Probe: A liveness probe. For the Controller Service, this performs an hourly DescribeAvailabilityZones call to verify networking and authentication.

Controller Service

Manages the lifecycle of EBS volumes at the AWS level:

• CreateVolume: Provisions a new EBS volume. If creating from a snapshot, it uses the provided snapshot ID.
• DeleteVolume: Deletes an existing volume. It succeeds if the volume is 'available' (not attached) or if the volume is not found. It returns an error if the volume is currently attached.
• ControllerPublishVolume: Attaches a volume to a node. It selects an unused device name and polls until the volume state is in-use.
• ControllerUnpublishVolume: Detaches a volume from a node using DetachVolume.
• ControllerExpandVolume & ControllerModifyVolume: Used to resize volumes or change attributes (IOPS, throughput, type). These calls are coalesced to prevent hitting AWS API cooldowns.
• CreateSnapshot / DeleteSnapshot / ListSnapshots: Manages EBS snapshots.

Node Service

Manages the volume on the specific Kubernetes node:

• NodeStageVolume: Finds the device, formats it if necessary, runs fsck if already formatted, and mounts it.
• NodePublishVolume: Performs a bind-mount of the volume.
• NodeUnstageVolume / NodeUnpublishVolume: Unmounts the volume.
• NodeExpandVolume: Resizes the filesystem if the underlying volume has been expanded.
• NodeGetVolumeStats: Returns usage statistics (available, total, used bytes and inodes).
• NodeGetInfo: Returns the AWS InstanceID and accessible topology (Availability Zone).

The AWS EBS CSI Driver is composed of two main components deployed on Kubernetes to manage the lifecycle of EBS volumes. It follows the Container Storage Interface (CSI) specification and uses a split architecture to separate Kubernetes resource management from AWS-specific logic.

• EBS CSI Controller (ebs-csi-controller): A Kubernetes Deployment that watches for storage resource changes (like PVCs). It interacts with the AWS EC2 APIs to create, attach, and delete volumes.
• EBS CSI Node (ebs-csi-node): A Kubernetes DaemonSet running on every node. It implements the Node Service RPCs, allowing the Kubelet to mount EBS volumes as block devices or filesystems within container workloads.

To remain CO-agnostic (Container Orchestrator agnostic), the core ebs-plugin container does not interface directly with the Kubernetes API. Instead, it works alongside Kubernetes CSI sidecar containers (such as csi-provisioner, csi-attacher, and csi-resizer) which handle the generic Kubernetes resource watching and translate those changes into standardized CSI RPC calls for the ebs-plugin.

Scalability tests are defined by the combination of CLUSTER_TYPE and TEST_TYPE.

CLUSTER_TYPE

Determines the cluster architecture and node provisioning strategy:

• pre-allocated: Creates additional worker nodes during setup. By default, it allocates 1 m7a.48xlarge EC2 instance for every 100 StatefulSet replicas.
• karpenter: Installs Karpenter during setup. Karpenter provisions and deletes worker nodes dynamically during the test run.

TEST_TYPE

Determines the specific workload being exercised:

• scale-sts: Scales a StatefulSet to $REPLICAS, waits for readiness, deletes the StatefulSet, and waits for PV deletion. Tests the full dynamic provisioning lifecycle.
• expand-and-modify: Creates $REPLICAS volumes and patches PVC capacity/VolumeAttributesClass at a rate of 5 PVCs per second. Tests ControllerExpandVolume and ControllerModifyVolume. Use MODIFY_ONLY=true or EXPAND_ONLY=true to isolate these behaviors.
• snapshot-volume-scale: Creates $REPLICAS volumes and takes $SNAPSHOTS_PER_VOLUME snapshots of each.
• volume-lifecycle-churn: Runs $WAVES sequential waves of $REPLICAS short-lived Jobs. Each wave's volumes are torn down while the next wave's volumes are created, simulating concurrent create/attach and detach/delete pressure (e.g., Spark pipelines).