DOCSEARCHbyAlgolia
Submit Docs

kube-state-metrics

repository·main·Indexed 27 days ago

https://github.com/kubernetes/kube-state-metrics

A service that listens to the Kubernetes API server and generates raw, unmodified metrics about the state of Kubernetes objects such as deployments, nodes, and pods. Designed for consumption by Prometheus or compatible scrapers, it exposes metrics via an HTTP endpoint and supports horizontal sharding, opt-in metrics, and custom allow/deny lists using ECMAScript regular expressions.

Tokens
23.8K
Snippets
30
Records
119
Agent score
90%

What's inside kube-state-metrics

  1. Overview of kube-state-metrics

    main

    kube-state-metrics (KSM) is a service that listens to the Kubernetes API server to generate metrics regarding the state of Kubernetes objects (e.g., deployments, nodes, and pods).

    Key characteristics:

    • Raw Data: KSM exposes raw, unmodified data from the Kubernetes API. Unlike kubectl, which applies heuristics to make messages more human-readable, KSM provides the exact values from the API so users can apply their own logic.
    • Metric Scope: It focuses on the state of objects within the cluster rather than the health of the Kubernetes components themselves.
    • Consumption: Metrics are exported as plaintext and are designed to be consumed by Prometheus or any Prometheus-compatible scraper.

  2. Understand Kube-State-Metrics metric design principles

    main
    Kube-State-Metrics follows Prometheus best practices for naming and labeling. The core design philosophy is to expose metrics at an individual object level and avoid pre-computation. Instead of providing derived metrics (e.g., kube_pod_total), the project provides raw metrics (e.g., kube_pod_info) so users can perform their own computations (e.g., count(kube_pod_info)) for maximum flexibility.

  3. Understand kube-state-metrics performance characteristics

    main

    Users running kube-state-metrics in large production clusters may encounter two primary performance issues:

    1. High Scrape Latency: The /metrics endpoint may take 10s - 20s to respond, which can cause Prometheus scrape timeouts and stale time series.
    2. High Memory Usage: The service maintains an in-memory representation of Kubernetes objects, which can lead to Out-Of-Memory (OOM) kills if Kubernetes resource limits are set too low.

    To mitigate these, ensure your Kubernetes resource limits account for the size of your cluster's object state and that your Prometheus scrape timeout is sufficient for the cluster scale.

  5. Monitor StatefulSet health and availability

    main

    Use these PromQL queries to track the rollout status and availability of your StatefulSets.

    • Percentage of updated replicas: Calculates how much of the rollout is complete.
    • Number of unavailable replicas: Calculates the count of replicas that are not currently available.
    # Percentage of updated replicas
(kube_statefulset_status_replicas_updated / kube_statefulset_replicas) * 100

# Number of unavailable replicas
kube_statefulset_replicas - kube_statefulset_status_replicas_available

  6. Expose PodDisruptionBudget annotations and labels as metrics

    main

    To include Kubernetes annotations or labels from PodDisruptionBudget resources as Prometheus labels, you must use the following CLI flags during kube-state-metrics startup:

    • Use --metric-annotations-allowlist to permit specific annotations to be converted into metrics with the prefix annotation_.
    • Use --metric-labels-allowlist to permit specific labels to be converted into metrics with the prefix label_.

    Note: These features are currently marked as EXPERIMENTAL.

  7. Expose ServiceAccount annotations and labels as metrics

    main

    To convert Kubernetes annotations or labels from ServiceAccounts into Prometheus labels, you must use the following CLI flags to allowlist them:

    • Use --metric-annotations-allowlist to enable specific annotations.
    • Use --metric-labels-allowlist to enable specific labels.

    When enabled, these will appear as annotation_<NAME> and label_<NAME> respectively in the kube_serviceaccount_annotations and kube_serviceaccount_labels metrics.

  8. Use a configuration file for kube-state-metrics

    main

    You can provide a YAML configuration file to define options. If the --config flag is used, the settings in the file will override command-line flags.

    To allow kube-state-metrics to start even if the config file is not yet available (e.g., waiting for a ConfigMap to be created), use the --continue-without-config flag. In this mode, kube-state-metrics will watch and reload the configuration once the file is created.

  10. Configure Custom Resource State Metrics

    main

    To generate metrics from Custom Resources (CRs), you must provide a configuration defining the resources and the fields to be converted into metrics. You can provide this configuration via two flags:

    1. --custom-resource-state-config: Accepts an inline YAML string.
    2. --custom-resource-state-config-file: Accepts a path to a YAML configuration file.

    Note on precedence and configuration:

    • If both flags are provided, the inline configuration (--custom-resource-state-config) takes precedence.
    • When using a config file, the equivalent YAML key is custom_resource_state_config_file. The deprecated key custom_resource_config_file is still supported but should be migrated.
    • If multiple entries for the same resource exist (including different API versions), kube-state-metrics will exit with an error.

    To run kube-state-metrics in custom-resource-mode only (ignoring all standard Kubernetes resources and only processing your configured CRs), set the flag --custom-resource-state-only=true.

    apiVersion: apps/v1
kind: Deployment
metadata:
  name: kube-state-metrics
  namespace: kube-system
spec:
  template:
    spec:
      containers:
      - name: kube-state-metrics
        args:
          - --custom-resource-state-config
          -  |
              kind: CustomResourceStateMetrics
              spec:
                resources:
                  - groupVersionKind:
                      group: myteam.io
                      version: "v1"
                      kind: Foo
                    metrics:
                      - name: active_count
                        help: "Count of active Foo"
                        each:
                          type: Gauge
                          ...
          - --custom-resource-state-only=true

  11. Identify static vs dynamic object properties in metrics

    main

    To manage metric complexity and cardinality, Kube-State-Metrics categorizes object properties into two types:

    1. Static Properties: Properties with a 1:1 relationship to the object that do not change during its lifecycle (e.g., name, namespace, uid). These are typically grouped into an _info metric.
    2. Dynamic Properties: Properties that change during the object's lifecycle (e.g., a Pod's status like Pending or Running). These are reported as part of a "State Set" that includes labels identifying the object alongside the dynamic property.

  12. Expose CertificateSigningRequest annotations and labels as metrics

    main

    By default, Kubernetes annotations and labels are not exported as Prometheus labels to prevent cardinality explosion. To include specific annotations or labels from CertificateSigningRequest resources as Prometheus labels, use the following CLI flags:

    • --metric-annotations-allowlist: Specify the annotations you want to convert to labels.
    • --metric-labels-allowlist: Specify the labels you want to convert to labels.

    When enabled, these will be available via the kube_certificatesigningrequest_annotations and kube_certificatesigningrequest_labels metrics respectively.