DOCSEARCHbyAlgolia
Submit Docs

Kubernetes Monitoring for Grafana Helm Charts

repository·main·Indexed 20 days ago

https://github.com/grafana/k8s-monitoring-helm

Helm charts for deploying Kubernetes Monitoring for Grafana. Includes documentation on the k8s-monitoring chart, migration from the deprecated v1 version, and configuration for features such as annotation-based autodiscovery for metrics scraping, application observability, service name alignment with OpenTelemetry semantic conventions, and span logs and metrics connectors.

Tokens
165.6K
Snippets
395
Records
550
Agent score
65%

What's inside k8s-monitoring-helm

  1. Understand the Kubernetes Monitoring Helm chart structure

    main

    The k8s-monitoring Helm chart is a parent chart that orchestrates multiple software packages, configurations, and secrets. It is composed of several key components:

    • Feature Charts: Individual features are implemented as sub-charts located in the charts/ directory. These are not standalone but are included as dependencies in the main k8s-monitoring chart and interact via template functions.
    • Collectors: These are Grafana Alloy instances deployed as Kubernetes workloads via the Alloy Operator. They are located in the collectors/ directory.
    • Destinations: Configuration examples and values.yaml files for various data destinations (where metrics, logs, traces, etc., are sent) are located in the destinations/ directory.
    • Examples: Full configuration and customization examples are provided in the examples/ directory to guide setup.

  2. What is Tail Sampling?

    main

    Tail sampling is a technique used to make sampling decisions for traces after all spans in a trace have been collected, rather than at the start of the trace (head sampling).

    This approach allows for more sophisticated observability strategies, such as:

    • Keeping all traces that contain errors, regardless of their duration.
    • Sampling traces based on total latency.
    • Making decisions based on specific attributes found anywhere within the trace.
    • Applying complex logic that combines multiple conditions.

  3. When to use Beyla span metrics only mode

    main

    Use the beyla.deliverTracesToApplicationObservability: false configuration when:

    • Cost Optimization: You want observability without the high cost of trace storage.
    • High Throughput: You are instrumenting services where full tracing would generate excessive data volume.
    • Metric-Only Requirements: You specifically need RED metrics (request rate, error rate, duration) and service graph relationships but do not require detailed trace analysis.
    • Hybrid Tracing: You already have tracing data from other sources and only need Beyla to provide metrics.

  5. What are Prometheus Operator Objects?

    main

    The Prometheus Operator Objects feature allows the monitoring stack to discover, process, and utilize specific Prometheus Operator resources. This enables automated metric scraping by defining how Prometheus should interact with Kubernetes resources.

    Supported object types include:

    • ServiceMonitor: Defines how to scrape metrics from Kubernetes Services.
    • PodMonitor: Defines how to scrape metrics from Kubernetes Pods.
    • Probe: Defines how to scrape metrics from prober exporters.

  6. How the Custom Processor works

    main

    The Custom Processor allows you to inject arbitrary Alloy configuration into a feature's pipeline. It works by handling specific (telemetry type, ecosystem) tuples. For every tuple you want to process, you must enable it, specify an input (where the chart forwards data to you), and provide a config (your Alloy pipeline).

    Key characteristics:

    • Multi-ecosystem support: A single processor can handle multiple types (e.g., Prometheus metrics and OTLP logs) simultaneously.
    • Isolation: The pipeline is emitted only in the collectors that use the specific feature, preventing cross-pollution.
    • Experimental: This feature is considered experimental and subject to change.

    To use it, you define a processor under dataProcessors and then reference that processor's name in the dataProcessors list of a specific feature (e.g., clusterMetrics).

    dataProcessors:
  my-processor:
    type: custom
    metrics:
      prometheus:
        enabled: true
        input: <input_ref>
        config: |
          # Your Alloy config here

clusterMetrics:
  enabled: true
  dataProcessors: [my-processor]

  8. How service name and namespace are detected for logs

    main

    The Pod Logs feature automatically assigns service.name and service.namespace resource attributes to collected logs.

    service.name detection

    By default, the detection chain for service.name is:

    1. The resource.opentelemetry.io/service.name pod annotation (highest precedence).
    2. The workload owner name (e.g., Deployment, StatefulSet).
    3. The pod name.
    4. The container name.

    To align with OpenTelemetry semantic conventions (and ensure consistency with Grafana Beyla metrics), set alignServiceNameWithOTelSemConv: true. The detection chain then becomes:

    1. resource.opentelemetry.io/service.name pod annotation.
    2. app.kubernetes.io/instance pod label.
    3. app.kubernetes.io/name pod label.
    4. Workload owner name.
    5. Pod name.
    6. Container name.

    When alignServiceNameWithOTelSemConv is enabled, service.version is also populated from the app.kubernetes.io/version pod label.

    service.namespace detection

    The service.namespace is set using the first available value from:

    1. The resource.opentelemetry.io/service.namespace pod annotation.
    2. The pod's namespace.

  9. How features work in Kubernetes Monitoring Helm

    main
    In the Kubernetes Monitoring Helm chart, features are implemented as subcharts. This architecture allows a feature to be enabled or disabled via a single flag in the parent chart's values.yaml, while encapsulating all feature-specific configurations within the subchart's own values.yaml file. This keeps the parent chart's configuration clean and manageable.

  10. Shard kube-state-metrics for scalability

    main

    When a Kubernetes cluster has a large number of objects, kube-state-metrics may struggle to keep up. Symptoms include scrape durations exceeding the scrape interval (e.g., > 60s), resource spikes in Alloy due to high metric volume, or kube-state-metrics failing to process objects efficiently.

    To resolve this, you can enable automatic sharding, which allows kube-state-metrics to distribute cluster resources across multiple replicas.

  11. Synchronize PrometheusRule objects with a Prometheus-compatible Ruler

    main

    A prometheus-type destination can automatically synchronize PrometheusRule objects with its Ruler API. When rules.enabled is set to true, the chart configures the Alloy mimir.rules.kubernetes component on the specified collector.

    Discovered recording and alerting rules are pushed to the destination's Mimir, Cortex, or Prometheus-compatible Ruler. This process reuses the destination's existing url, tenant ID, and authentication settings.

    Key configuration requirements:

    • rules.collector: Specifies which collector runs the synchronization loop. You must ensure exactly one collector is assigned to avoid race conditions between Alloy replicas when writing to the Ruler API.
    • rules.address: The base URL of the Ruler API. Note that if your url is a remote-write push URL (e.g., Grafana Cloud's /api/prom/push endpoint), you must set rules.address to the Mimir base URL instead of the specific push path.
    destinations:
  mimir:
    type: prometheus
    url: http://mimir.mimir.svc:9009/api/v1/push
    rules:
      enabled: true
      address: http://mimir.mimir.svc:9009/
      collector: alloy

  12. What are Processors and how do they work?

    main

    Processors are middleware used to transform telemetry data after it is gathered by a feature but before it reaches a destination. They can be used to rewrite labels, drop fields, redact values, or batch records.

    Core Concepts

    • Middleware Chain: A feature can opt into a chain of processors. The chart renders a pipeline and wires it between the feature and its destinations.
    • Tuple-based Resolution: Processors support specific (telemetry type, ecosystem) tuples. A feature can only attach a processor if both the feature and the processor support the same tuple. If a processor in a chain does not support a specific tuple, it is automatically dropped from the chain for that tuple.
    • Data Isolation: The chart uses a selected_destinations stamp to ensure that even when multiple features share a processor, data from different features does not cross-pollinate if they are destined for different locations.
    • Zero Cost for Unused Features: If a feature has no applicable processors for a specific tuple, the chart bypasses the processor logic and connects the feature directly to its destination receivers.