Grafana Community Helm Charts

Most fields in a Kubernetes StatefulSet are immutable. Changing certain configuration values in your values.yaml will not take effect via a standard helm upgrade unless specific conditions are met.

Immutable Fields (Requires manual intervention or recreation)

The following fields cannot be updated via standard upgrade and are not covered by the experimental recreate job:

• *.persistence.storageClass
• *.persistence.claims[].storageClass
• *.persistence.accessModes
• *.persistence.selector
• *.persistence.volumeAttributesClassName
• *.persistence.annotations
• Chart-generated selector labels

Fields covered by the Recreate Job

If you enable the experimental recreate job, you can update certain fields. The job deletes the StatefulSet with --cascade=orphan to allow the new spec to take effect.

Supported updates via Recreate Job:

• *.persistence.size (and cache persistence.storageSize)
• *.podManagementPolicy
• serviceName (affected by nameOverride or fullnameOverride)
• Volume-claim count, claim name, or claim size (if defaults.statefulSetRecreateJob.patchPVC is true)

Note: The recreate job does not work for Memcached cache StatefulSets.

In a standard configuration, the live-store is a single flat StatefulSet where each Kafka partition has exactly one live-store instance. If that pod fails, recent-trace queries for that partition fail.

Zone-aware replication solves this by rendering one StatefulSet per zone. Each zone runs a full set of live-stores and consumes Kafka partitions independently. This ensures that the loss of a single zone does not break recent-trace queries.

Key requirements and behaviors:

• Rollout Operator: The chart requires a rollout_operator to coordinate rollouts and manage ZoneAwarePodDisruptionBudget. The chart will refuse to render without it.
• Replicas: liveStore.replicas must equal the Kafka partition count. This value represents the size of one zone, not the total count across all zones.
• Topology: Use topologyKey (e.g., topology.kubernetes.io/zone) to ensure anti-affinity. At least 2 zones are required.
• Switching on: Enabling zone-awareness replaces the flat StatefulSet with per-zone ones. This causes all live-stores to restart and requires them to replay the Kafka lookback period to rebuild state. Plan this as a full live-store restart.

rollout_operator:
  enabled: true

liveStore:
  enabled: true
  replicas: 3  # partitions per zone; must equal the Kafka partition count
  zoneAwareReplication:
    enabled: true
    topologyKey: topology.kubernetes.io/zone
    zones:
      - name: zone-a
        nodeSelector:
          topology.kubernetes.io/zone: eu-west-1a
      - name: zone-b
        nodeSelector:
          topology.kubernetes.io/zone: eu-west-1b

When sidecar.datasources.enabled is set to true, an init container is deployed. It watches Secrets (recommended) or ConfigMaps for resources matching the label in sidecar.datasources.label. The files are written to a folder and imported into Grafana on startup.

Reloading Datasources: To reload datasources automatically when a Secret changes, set sidecar.datasources.skipReload: false and provide a sidecar.datasources.reloadURL (e.g., http://<svc-name>.<namespace>.svc.cluster.local/api/admin/provisioning/datasources/reload).

Example Secret for Postgres Datasource

apiVersion: v1
kind: Secret
metadata:
  name: grafana-datasources
  labels:
    grafana_datasource: 'true'
stringData:
  pg-db.yaml: |
    apiVersion: 1
    datasources:
      - name: My pg db datasource
        type: postgres
        url: my-postgresql-db:5432
        user: db-readonly-user
        secureJsonData:
          password: 'SUperSEcretPa$$word'
        jsonData:
          database: my_datase
          sslmode: 'disable'
        editable: false

The tempo-distributed Helm chart installs a set of core components by default, while other specialized components are optional and must be explicitly enabled via their respective enabled flags.

Always Installed:

• distributor: Writes spans to Kafka in Tempo 3.0.
• querier: Handles query execution.
• query-frontend: Manages query requests.

Optional Components: To use these, you must set their enabled key to true in your values.yaml:

• backend-scheduler (backendScheduler.enabled): Required for compaction and retention.
• backend-worker: Automatically enabled when backend-scheduler is active; executes compaction jobs.
• block-builder (blockBuilder.enabled): Consumes from Kafka and writes blocks to object storage.
• live-store (liveStore.enabled): Consumes from Kafka and serves recent-data queries.
• metrics-generator: Generates metrics from traces.
• memcached: Provides caching.
• memcached-exporter: Exports metrics for Memcached.
• gateway: Provides an entry point for requests.
• rollout-operator: Manages rollout processes.

# Example enabling optional components
backendScheduler:
  enabled: true
blockBuilder:
  enabled: true
liveStore:
  enabled: true
metricsGenerator:
  enabled: true

The built-in MinIO subchart has been removed in Tempo 3.0. Setting minio.enabled: true will now cause a render failure.

Option 1: Migrate to external S3 storage

Deploy MinIO (or another S3-compatible store) separately and configure Tempo via storage.trace.s3.

Option 2: Keep existing built-in MinIO running (Manual Detachment)

If you used minio.enabled: true, a standard helm upgrade will delete the MinIO resources and your data. To prevent this, you must annotate the MinIO resources with helm.sh/resource-policy: keep to detach them from the Helm release before upgrading.

1. Annotate resources:

# RELEASE and NAMESPACE are your existing tempo-distributed install.
for kind in deployment service secret pvc serviceaccount; do
  kubectl -n "$NAMESPACE" annotate "$kind" \
    -l app=minio,release="$RELEASE" \
    helm.sh/resource-policy=keep --overwrite
done

2. Verify annotations (especially on the PVC):

kubectl -n "$NAMESPACE" get deploy,svc,secret,pvc,sa -l app=minio,release="$RELEASE" \
  -o jsonpath='{range .items[*]}{.kind}/{.metadata.name}: {.metadata.annotations.helm\.sh/resource-policy}{"\n"}{end}'

3. Update values: Remove the minio: block and point Tempo to the surviving MinIO service using the storage.trace.s3 configuration.