etcd-operator

On every reconciliation, the EtcdClusterReconciler compares the live state of a member's Pod against the desired configuration. The following table defines the actions taken based on the observed state:

Note: Every Pod creation (except for a healthy, matching Pod) increments the RecreateCount. A CORRUPT alarm on a member overrides these heuristics and forces the member directly into the Replacing phase.

In version 0.3.0, the etcd-operator introduces a redesigned reconciliation workflow centered around a new Custom Resource Definition (CRD) called EtcdMember.

Core Abstractions

• EtcdCluster: The top-level resource representing the entire etcd cluster.
• EtcdMember: A new namespaced CRD representing exactly one etcd member/Pod slot (ordinal). It serves two purposes:
  1. Durable State Store: It records per-member retry and lifecycle state in EtcdMember.Status, allowing the operator to remember progress (e.g., retry counts) across controller restarts.
  2. Managed Child Resource: It owns the underlying Pod, PVC, and TLS certificates.

Reconciliation Logic

• Single Reconciler: The EtcdClusterReconciler remains the sole controller, but it now manages both EtcdCluster and EtcdMember resources.
• Sequential Orchestration: To ensure stability, the operator follows a strict rule: exactly one action is taken at a time per cluster. All other orchestration actions (such as scaling, upgrades, or configuration updates) are blocked until every existing member reaches a Ready state.
• Self-Healing: The design enables resumable remediation for cluster-wide issues like CORRUPT or NOSPACE alarms, as well as total quorum loss. Because state is stored in EtcdMember.Status, the operator can track how many times it has attempted to fix a specific member and can decide to replace the member if retries are exhausted.

The etcd-operator manages etcd clusters using a single reconciler (EtcdClusterReconciler) that orchestrates two Custom Resource Definitions (CRDs): EtcdCluster (the top-level cluster object) and EtcdMember (the individual member object).

Key architectural concepts:

• Single Reconciler Pattern: Instead of having separate controllers for members and clusters, a single reconciler manages both. Any change to an owned resource (like a Pod, PVC, or EtcdMember) triggers a reconciliation of the parent EtcdCluster.
• Durable Member State: EtcdMember acts as a durable state store for per-member information (like Status.Phase and RecreateCount) that survives controller restarts.
• Managed Child Resources: Each EtcdMember owns its specific Pod, PersistentVolumeClaim, and TLS certificate Secret. This allows Kubernetes to handle garbage collection and ordered cleanup (e.g., using finalizers to ensure a member is removed from etcd before its PVC is deleted).
• Readiness Gate: To ensure stability, the operator implements a readiness gate. It will not perform high-level policy actions (like scaling, upgrading, or updating configuration) if any existing EtcdMember is not in the Ready phase. However, it will continue to perform per-member repairs, alarm remediation, and quorum recovery even if members are not ready.

The EtcdMember lifecycle is managed through a Status.Phase state machine. The operator follows a strict rule: always write Status.Phase before performing the mutation it describes to ensure safety during crashes.

Phase: Provisioning

Used when a member is joining the cluster. The sequence is:

1. Create TLS Secret and PVC.
2. If ordinal 0 and first member: bootstrap directly.
3. Otherwise: call MemberAdd(learner).
4. Start the Pod (using --initial-cluster-state=new for bootstrap, existing otherwise).
5. Once the Pod is healthy and the member is promoted from learner to voting member, the phase advances to Ready.

Phase: Ready

An EtcdMember is only considered Ready if:

• The etcd process is healthy.
• The member has been promoted from a learner to a voting member via MemberPromote.

Phase: Replacing

Used during repairs or upgrades. If a member in Provisioning or Recreating fails to become healthy after repeated retries, it escalates to Replacing.

The etcd-operator uses a specific priority-based reconciliation loop to manage etcd cluster health and member lifecycles. When the operator reconciles, it evaluates conditions in a strict order to decide which action to take. This ensures that critical issues (like manual pauses or lost quorum) are addressed before routine repairs or scaling operations.

Reconciliation Priority Order:

1. Spec.Paused is set: If the cluster is paused, the operator does nothing else. This allows for manual human intervention without automation interference.
2. Continue lost-quorum recovery: If Status.QuorumRecovery is not nil, the operator continues the existing recovery process. This is exempt from readiness gates because a cluster without quorum cannot make progress on other tasks.
3. Clean up Terminating members: If an EtcdMember has a DeletionTimestamp set, the operator continues its leave sequence (cleaning up Pods, PVCs, and certificates).
4. CORRUPT alarm remediation: If a CORRUPT alarm is detected on a member, that member is immediately forced to Phase: Replacing.
5. NOSPACE alarm remediation: If a NOSPACE alarm is active, the operator performs compaction/defragmentation. This can preempt repairs on other members.
6. Per-member repair: If a Ready member is unhealthy, the operator picks exactly one (typically the lowest ordinal or a non-leader) and moves it to Phase: Replacing to begin the repair ladder.
7. Start lost-quorum recovery: If the cluster cannot serve linearizable requests and no other remedies (steps 1-6) have worked, the operator begins a lost-quorum recovery. This is a last resort as it discards all existing member data.
8. Advance non-Ready members: The operator works on members in Pending, Provisioning, or Replacing phases. It prioritizes promoting Learner members to ensure they catch up before being promoted.
9. Cluster operations: Once all members are Ready and healthy, the operator proceeds with update-config, scale, or upgrade operations.

The operator follows a specific hierarchy and logic when repairing members to maintain cluster stability:

1. Readiness Gate: The operator will not perform any orchestration (scale, upgrade, config updates) if any existing EtcdMember is not in a Ready state. It waits for the current member to become ready before moving to the next.
2. Repair Strategy:
   • Level 1 (Recreate): The default remedy is to delete and recreate the Pod with the latest config from EtcdCluster.Spec (excluding Version).
   • Level 2 (Replace): If three failed recreate attempts occur, the operator falls back to removing the etcd member, cleaning up its data and TLS certificate, and adding it back as a new member with a fresh Pod.
3. Leadership Transfer: Before recreating or removing a Pod that is currently the etcd leader (during scale-in, upgrades, or repairs), the operator moves leadership to the existing member with the lowest ordinal (excluding the member being acted upon).
4. Concurrency: The operator fixes broken members one at a time, except when handling urgent NOSPACE, CORRUPT, or lost-quorum recovery scenarios.

TLS configuration is managed via the tls field in EtcdClusterSpec, which uses a TLSCertificate object. You must choose a provider and provide a corresponding providerCfg (ProviderConfig).

ProviderConfig

ProviderConfig allows you to choose between two certificate providers:

1. autoCfg (ProviderAutoConfig): Uses the operator's internal mechanism to generate certificates.
2. certManagerCfg (ProviderCertManagerConfig): Integrates with cert-manager to issue certificates.

Common Certificate Settings

Both providers share several configuration fields:

• commonName: The X509 certificate subject attribute (max 64 characters).
• organizations: An array of Organization names for the certificate.
• altNames: Contains dnsNames (defaults to $(POD_NAME).$(ETCD_CLUSTER_NAME).$(POD_NAMESPACE).svc.cluster.local if empty) and ipAddresses to be added to the certificate's Subject Alternative Names.
• validityDuration: Human-readable duration (e.g., 100d12h). Defaults to 90d for cert-manager and 365d for auto.
• caBundleSecret: The name of the secret containing the CA bundle used by pods to verify TLS communications.

tls:
  provider: "cert-manager"
  providerCfg:
    certManagerCfg:
      issuerKind: "ClusterIssuer"
      issuerName: "my-etcd-issuer"
      validityDuration: "180d"
      commonName: "etcd-cluster"

The etcd-operator manages member health and cluster state through a prioritized reconciliation loop. When the operator reconciles, it executes specific phases based on the current state of the EtcdCluster and its EtcdMembers.

Priority Order (High to Low):

1. Finalize cluster: Triggered if EtcdCluster.DeletionTimestamp is set. This short-circuits all other logic.
2. Pause: If Spec.Paused is set, no other logic runs.
3. Continue lost-quorum recovery: Only if Status.QuorumRecovery != nil.
4. Clean up Terminating members: Processes members with a DeletionTimestamp.
5. Alarm Remediation: Handles CORRUPT (replaces member) or NOSPACE (compact/defragment) alarms.
6. Per-member repair: Recreates unhealthy Ready members.
7. Start lost-quorum recovery: A last-resort mechanism that discards all member data.
8. Learner/Not-Ready advancement: Promotes learners or advances members in Pending/Provisioning/Replacing states.
9. Configuration/Scaling/Upgrades: These phases (Update config, Scale out/in, Upgrade) only run once all existing EtcdMembers are Ready.

Key Concept: The Readiness Gate Operations like scaling, upgrading, or updating configuration are gated. They will not execute until the operator confirms that every existing member in the cluster has reached the Ready state.

The v0.3.0 design addresses limitations in the v0.1.0/v0.2.0 implementations regarding durability and self-healing: