Garage Kubernetes Operator

The Garage Kubernetes Operator manages Garage, a distributed, self-hosted, S3-compatible object storage system. It provides a declarative way to manage the entire cluster lifecycle, including storage and gateway tiers, bucket management, and multi-cluster federation.

Key Capabilities:

• Declarative Lifecycle: Manage StatefulSets, configurations, and layouts via Custom Resources (CRDs).
• Unified Tiers (v1beta2): Combine storage pods and persistent-identity S3 gateways in a single GarageCluster resource.
• Node-local Pools: Bind Garage identities to specific Kubernetes Nodes and HostPath disks (requires Kubernetes 1.27+).
• Bucket & Key Management: Provision buckets, quotas, and S3 credentials using kubectl.
• Multi-cluster Federation: Automatically discover nodes across different Kubernetes clusters.
• Scale Subresource: Supports kubectl scale for auto-managed storage groups.
• COSI Driver: Optional Kubernetes-native object storage provisioning.

Node-local pools are only supported in v1beta2 of the API.

v1beta1 Compatibility

A v1beta1 view is maintained for backward compatibility. It keeps the default StatefulSet/PVC group editable and carries the pool list in a reserved conversion annotation:

garage.rajsingh.info/v1beta2-node-local-pools

When converting back to v1beta1, only the pool list is restored; the v1beta2-only marker is removed. The v1beta1 webhook protects against accidental erasure by rejecting mutations to the payload if the original request was made via v1beta1.

Migration Warning for Unreleased Prototypes

If you are using unreleased prototypes with the following keys, you must stop them and recreate or migrate them before installing the final CRD to avoid identity conflicts:

• storage.workload: DaemonSet
• storage.pools
• storage.nodePools
• backing: StoragePool
• backing: NodePool
• poolName
• nodePoolName
• storage-pool/node-pool labels
• conversion annotations used in prototypes

The GarageCluster controller implements a safety mechanism to prevent unsafe layout mutations during storage rollouts. When a rollout is in progress, the controller may enter a state where it waits for coordination with other nodes.

Key Behaviors:

• Rollout Wait State: If the controller encounters errLayoutMutationPending, it treats this as a retryable safety wait rather than a terminal failure. It will publish StorageRolloutReady=False with the reason Waiting.
• Health Projection: The operator clears the previous Status.Health observation before each new status pass. This ensures that if the Garage Admin API becomes unavailable, the operator does not incorrectly report a stale "healthy" status.
• Rollout Restrictions: While a storage rollout boundary is active, the operator restricts operations to prevent workload outages. It will not import roles or stage/apply layout mutations until the rollout proof completes.
• Allowed Operations: During an active rollout wait, only the following are permitted:
  1. Federation RPC reconnect work.
  2. Explicitly requested connect-nodes operations.

Operational Recovery

The operator does not automatically perform high-risk recovery actions. It will not clear a Garage layout guard, invoke skip-dead-nodes, delete Pods, or infer that a disconnected node is safe to remove. These require separate operational supervision.

The spec.deletionPolicy field determines how the remote Garage bucket is handled when the GarageBucket Kubernetes resource is deleted:

• Delete (default): The operator attempts to delete the Garage bucket. This only succeeds if the bucket is empty. If the bucket contains objects, the resource will stay in a Deleting state and set DeletionBlocked=True with the reason BucketNotEmpty. You must manually remove objects/multipart uploads via an S3 client to allow deletion.
• Retain: The Kubernetes resource is removed, but the bucket, its data, aliases, and settings remain in Garage.

Important: If using Retain, record the status.bucketId before deleting the CR so you can re-attach to it later using spec.bucketId.

apiVersion: garage.rajsingh.info/v1beta1
kind: GarageBucket
metadata:
  name: postgres-backups
  namespace: storage
spec:
  clusterRef:
    name: external-garage
  deletionPolicy: Retain

The GarageReferenceGrant resource allows administrators to define trust boundaries for cross-namespace access to Garage resources.

How it works

The grant lives in the destination namespace. It uses spec.from to list allowed source resources.

Configuration

• spec.from: A list of source entries. Each entry contains a source kind (GarageBucket, GarageKey, or GarageAdminToken) and either an exact namespace or a namespaceSelector (using standard Kubernetes matchLabels or matchExpressions).
• spec.to: Narrows the destination. If omitted, it allows all resources of the target kind.
  • Omitting spec.to defaults to allowing GarageCluster and GarageBucket targets.
  • To allow GarageKey targets, you must explicitly add an entry in spec.to.

Important Constraints

• Namespace Labels: If using namespaceSelector, access is granted to any namespace that carries the matching label. Ensure these labels are controlled by a trusted policy.
• No Cross-Namespace Admin Tokens: While GarageAdminToken is in the schema for compatibility, a grant cannot make static admin credentials cross-namespace.
• No Cross-Namespace Nodes: GarageNode is never allowed cross-namespace.
• Cleanup: Before removing a grant, check status.inUseBy. It is safe to remove a grant only when this list is empty. Removing a grant does not revoke already-issued Garage permissions; it only prevents future reconciliations from succeeding.

When a nodeLocalPool is active, the operator manages several resources to ensure safe membership transitions and identity stability.

Managed Resources for a pool named local-700 in cluster garage:

• DaemonSet: garage-storage-local-700.
• ConfigMap: Active revision garage-storage-local-700-config-<config-hash>.
• Internal GarageNode: One per active Kubernetes Node (with backing: NodeLocalPool). This resource manages the Garage layout role but does not create a StatefulSet or PVC.
• Activation Gate: A scheduler-enforced gate on every DaemonSet Pod to prevent premature scheduling during membership changes.
• Labels & Annotations:
  • garage.rajsingh.info/node-local-pool=<pool name>
  • garage.rajsingh.info/kubernetes-node=<node name or stable hash>

Note: Node-local pool-backed GarageNode resources cannot be created manually by users; the admission webhook prevents forged ownership.

The operator manages several Custom Resources (CRDs) under the API group garage.rajsingh.info. Each resource has specific responsibilities and manages corresponding Kubernetes artifacts: