Grafana Rollout Operator

The operator coordinates the rollout of pods across multiple StatefulSets that belong to the same rollout group.

Configuration Requirements

To participate in a rollout group, StatefulSets must meet these criteria:

1. Labeling: Must have a rollout-group label. The value of this label identifies the group (e.g., rollout-group: ingester).
2. Pod Identification: The spec.template must include a name label so the operator can identify pods belonging to that specific StatefulSet.
3. Update Strategy: All StatefulSets in the group must use the OnDelete update strategy. If any member uses a different strategy, the operator will skip the group and log an error.

Rollout Guarantees

For each rollout group, the operator ensures:

• Sequentiality: Pods in two different StatefulSets within the same group are never rolled out at the same time.
• Stability: A StatefulSet only begins its rollout if all pods in all other StatefulSets in the same group are Ready.
• Availability Control: The number of non-Ready pods is limited by the rollout-max-unavailable annotation (defaults to 1).
  • 1: Pods roll out sequentially.
  • > 1: Pods roll out in parallel up to the configured limit.
  • Note: Outdated pods in unrecoverable states (e.g., CrashLoopBackOff) are deleted even if the limit is reached to allow the rollout to proceed.

Max Unavailable Configuration

Use the rollout-max-unavailable annotation to control concurrency. Values <= 0 are invalid and will default to 1 with a warning.

# Example StatefulSet configuration for a rollout group
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: ingester-zone-a
  labels:
    rollout-group: ingester
  annotations:
    rollout-max-unavailable: "2"
spec:
  updateStrategy:
    type: OnDelete
  template:
    metadata:
      labels:
        name: ingester-zone-a

ZoneAwarePodDisruptionBudget (ZPDB) is a custom resource designed for StatefulSets that span multiple logical zones. Unlike a standard PodDisruptionBudget which evaluates availability across all pods globally, ZPDB evaluates availability within specific zones to ensure that maintenance in one zone does not disrupt availability in others.

Core Eviction Logic

A pod eviction is only allowed if:

1. The number of unavailable pods in that specific zone is within the maxUnavailable threshold.
2. No other zone is currently experiencing a disruption.

Partition Awareness

ZPDB supports partition awareness, which is useful for workloads (like Mimir ingesters) where availability is required per partition rather than per zone. In this mode, the operator determines the partition for a pod and evaluates the eviction against the unavailable counts for all pods serving that specific partition across all zones.

A pod eviction is only allowed if the number of unavailable pods serving a specific partition is less than the maxUnavailable value.

Assumption: This mode assumes there is only one pod per partition per zone.

You can coordinate the scaling of StatefulSets in a group to ensure stability across zones (e.g., scaling up a follower zone only after the leader zone is stable).

Implementation Steps

1. Set a Leader: On the follower StatefulSet, use the grafana.com/rollout-downscale-leader annotation to point to the leader StatefulSet name.
2. Set Scaling Delay: On all participating StatefulSets, set grafana.com/min-time-between-zones-downscale to a duration (e.g., 12h).
3. Ensure Stability: On followers, set grafana.com/rollout-upscale-only-when-leader-ready=true to prevent scaling up until the leader's replicas are all Ready.
4. Prepare for Downscale (Optional): To allow services to gracefully shut down, use the following:
   • Label: grafana.com/prepare-downscale=true
   • Annotation: grafana.com/prepare-downscale-http-path (the endpoint path)
   • Annotation: grafana.com/prepare-downscale-http-port (the service port)

Example Configuration for a Follower Zone

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: ingester-zone-b
  labels:
    grafana.com/prepare-downscale: "true"
  annotations:
    grafana.com/min-time-between-zones-downscale: "12h"
    grafana.com/rollout-downscale-leader: "ingester-zone-a"
    grafana.com/rollout-upscale-only-when-leader-ready: "true"
    grafana.com/prepare-downscale-http-path: "ingester/prepare-shutdown"
    grafana.com/prepare-downscale-http-port: "80"

The operator can mirror the replica count from a custom resource (the "source of truth") to a target StatefulSet. This is useful when using an HorizontalPodAutoscaler (HPA) that updates a custom resource instead of the StatefulSet directly.

Configuration Annotations

Set these on the target StatefulSet:

• grafana.com/rollout-mirror-replicas-from-resource-name: Name of the reference resource.
• grafana.com/rollout-mirror-replicas-from-resource-kind: Kind of the reference resource.
• grafana.com/rollout-mirror-replicas-from-resource-api-version: API version of the reference resource.
• grafana.com/rollout-mirror-replicas-from-resource-write-back: Set to false to prevent the operator from updating the reference resource's status subresource with the target's current replica count.

RBAC Requirements

The operator requires permissions to get and patch the status and scale subresources of the reference resource.

Example RBAC for a custom resource replica-templates:

- apiGroups:
  - rollout-operator.grafana.com
  resources:
  - replica-templates/scale
  - replica-templates/status
  verbs:
  - get
  - patch

The /admission/prepare-downscale webhook calls a downscale preparation endpoint on pods before decreasing replicas in objects labeled with grafana.com/prepare-downscale: true.

Required Object Labels and Annotations

To use this webhook, the target object must have:

• Label: grafana.com/prepare-downscale
• Annotations:
  • grafana.com/prepare-downscale-http-path
  • grafana.com/prepare-downscale-http-port

Behavior

• Upscaling: Requests to increase replicas or those that do not change the replica count are automatically approved.
• Downscaling Logic: For each pod being downscaled, the operator calls the endpoint defined by the path and port annotations. If any of these requests fail, the entire downscaling request is rejected.
• Time-based Throttling: If the annotation grafana.com/last-downscale is present, the operator checks the time elapsed. If the difference is less than the value in grafana.com/min-time-between-zones-downscale (which can also be provided as a label), the request is rejected. The operator adds the grafana.com/last-downscale annotation to the stateful set upon successful validation.

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  labels:
    grafana.com/inject-rollout-operator-ca: "true"
    grafana.com/namespace: default
  name: prepare-downscale-default
webhooks:
- admissionReviewVersions:
  - v1
  clientConfig:
    service:
      name: rollout-operator
      namespace: default
      path: /admission/prepare-downscale
      port: 443
  failurePolicy: Fail
  matchPolicy: Equivalent
  name: prepare-downscale-default.grafana.com
  rules:
  - apiGroups:
    - apps
    apiVersions:
    - v1
    operations:
    - UPDATE
    resources:
    - statefulsets
    - statefulsets/scale
  scope: Namespaced
  sideEffects: NoneOnDryRun
  timeoutSeconds: 10

Both ValidatingAdmissionWebhook and MutatingAdmissionWebhook require a TLS connection. You can use an existing CA or use the operator's built-in self-signed certificate generator.

Using the Self-Signed Certificate Generator

By default, the operator generates a self-signed certificate and stores it in a Kubernetes Secret.

• Configuration Flags:
  • -server-tls.enabled=true: Enables the HTTPS server.
  • -server-tls.cert-file: Path to the certificate file.
  • -server-tls.key-file: Path to the key file.
  • -server-tls.self-signed-cert.secret-name: The name of the secret where the generated certificate is stored.
  • -server-tls.self-signed-cert.expiration: Sets the expiration duration (default is 1 year).

Automatic CA Bundle Injection

The operator can automatically inject the CA certificate into your webhook configurations. It looks for ValidatingWebhookConfiguration or MutatingWebhookConfiguration objects that have these labels:

• grafana.com/inject-rollout-operator-ca: "true"
• grafana.com/namespace: <the-operator-namespace>

Note: This injection happens at startup. If you add a new webhook configuration, you must restart the rollout-operator to trigger the injection. You can disable this feature with -webhooks.update-ca-bundle=false.

Required RBAC for Certificate Management

To use the self-signed generator and CA injection, the operator needs permissions to manage secrets and patch webhook configurations.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: rollout-operator-webhook-role
  namespace: default
rules:
  - apiGroups: [""]
    resources: [secrets]
    verbs: [create]
  - apiGroups: [""]
    resources: [secrets]
    resourceNames: [rollout-operator-self-signed-certificate]
    verbs: [update, get]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: rollout-operator-webhook-default-clusterrole
rules:
- apiGroups: [admissionregistration.k8s.io]
  resources: [validatingwebhookconfigurations]
  verbs: [list, patch, watch]

To run an instance of the rollout-operator locally for development, follow these steps to build the image, apply manifests, and access the services via port-forwarding.

1. Build the image: Run make build-image.
2. Apply manifests: Run ./apply.sh from the development directory.
3. Port forward services:
   • Operator HTTP: kubectl --namespace=rollout-operator-development port-forward svc/rollout-operator 8080:80 (Access at http://localhost:8080)
   • Operator Webhooks (HTTPS): kubectl --namespace=rollout-operator-development port-forward svc/rollout-operator 8443:443 (Access at https://localhost:8443)
   • Jaeger UI: kubectl --namespace=rollout-operator-development port-forward svc/jaeger 16686:16686 (Access at http://localhost:16686)

Note: If using Minikube or Docker Desktop with locally built images, ensure you use the Minikube Docker Daemon and set imagePullPolicy: Never in your manifests.

# Build and deploy
make build-image

cd development
./apply.sh

# Port forward
kubectl --namespace=rollout-operator-development port-forward svc/rollout-operator 8080:80
kubectl --namespace=rollout-operator-development port-forward svc/rollout-operator 8443:443
kubectl --namespace=rollout-operator-development port-forward svc/jaeger 16686:16686

To test how the operator handles pod evictions and ZPDB enforcement, you can simulate a node drain.

1. Watch pod status in a continuous loop:

   while true; do kubectl get pods -n rollout-operator-development; sleep 1; clear; done

2. Issue a drain in a separate shell:

   kubectl drain --pod-selector rollout-group=test-app minikube &; sleep 5; kubectl uncordon minikube

Important: The uncordon command is required in local single-node environments (like Minikube) to allow pods from the first zone to be re-deployed. Without re-deployment, the operator will not proceed to evict pods in the next zone.

Manually test the pod eviction webhook: You can trigger the admission webhook manually using curl to verify it processes Eviction requests correctly:

curl --insecure -X POST "https://127.0.0.1:8443/admission/pod-eviction" \
  -H "Content-Type: application/json" \
  --data '{...}'

# Watch pod status
while true; do kubectl get pods -n rollout-operator-development; sleep 1; clear; done

# Issue a drain
kubectl drain --pod-selector rollout-group=test-app minikube &; sleep 5; kubectl uncordon minikube

# Manual webhook test
curl --insecure -X POST "https://127.0.0.1:8443/admission/pod-eviction" \
  -H "Content-Type: application/json" \
  --data '{ "apiVersion": "admission.k8s.io/v1", "kind": "AdmissionReview", "request": { "uid": "test-eviction-123", "kind": {"group": "policy", "version": "v1", "kind": "Eviction"}, "resource": {"group": "policy", "version": "v1", "resource": "evictions"}, "name": "test-app-zone-a-0", "namespace": "rollout-operator-development", "operation": "CREATE", "subResource": "eviction", "userInfo": {"username": "test-user"}, "dryRun": false, "object": { "apiVersion": "policy/v1", "kind": "Eviction", "metadata": {"name": "test-app-zone-a-0", "namespace": "rollout-operator-development"}, "deleteOptions": {"gracePeriodSeconds": 30} } } }'