Understand the Alerting Flow
mainAlerting in the local observability stack follows a continuous pipeline to transform trace data into proactive notifications:
- Spans arrive at Tempo: Application trace data is ingested.
- Metrics Generator computes RED metrics: Calculates Rate, Errors, and Duration.
- Metrics written to Prometheus: Metrics are stored as time series.
- Grafana evaluates alert rules: Periodically checks conditions against Prometheus data.
- Alertmanager sends notifications: Triggers alerts when conditions are met.
Available Metrics for Alerting: These metrics are automatically generated from traces and can be queried via the Prometheus datasource in Grafana:
| Metric | Type | Use Case |
|---|---|---|
traces_spanmetrics_latency_bucket | histogram | Percentile latency alerts (p95, p99) |
traces_spanmetrics_calls_total | counter | Request rate, error rate alerts |
traces_spanmetrics_size_total | counter | Throughput alerts |