DOCSEARCHbyAlgolia
Submit Docs

PromQL Anomaly Detection

repository·main·Indexed 19 days ago

https://github.com/grafana/promql-anomaly-detection

A framework for performing anomaly detection on time series data directly within Prometheus/PromQL using recording and alerting rules. It generates adaptive anomaly bands (short-term, long-term, and margin) to detect threshold crossings without requiring external systems. Supports multiple detection strategies, including 'adaptive' for normally distributed metrics and 'robust' for spiky data, and provides a Docker-based demo environment including Grafana and OpenTelemetry.

Tokens
3.2K
Snippets
7
Records
13
Agent score
63%

What's inside promql-anomaly-detection

  2. How the anomaly detection framework works

    main

    The framework operates using two main components:

    1. Recording Rules

    These rules generate upper and lower bounds (anomaly bands) for each time series. The bands are composed of:

    • Short term bands: Expand based on variability observed over a period (typically 24-26 hours).
    • Long term bands: Incorporate seasonality to adapt to daily or weekly recurrent patterns.
    • Margin bands: Provide a minimum band width when observed variability is too low.

    2. Alerting Rules

    These rules monitor the time series and trigger when a value crosses the generated anomaly bands for a significant period of time.

  3. Tag metrics for anomaly detection

    main

    The framework identifies metrics for anomaly detection using specific labels. You must use recording rules or relabeling rules to apply these labels to your existing metrics.

    Required Labels

    • anomaly_name: The unique name of the anomaly metric. This is used to identify the metric uniquely.

    Optional Labels

    • anomaly_strategy: Selects the detection algorithm. Supported values:
      • adaptive (default): Uses mean and standard deviation with a 26h smoothing function and high pass filter. Best for normally distributed metrics and quick detection of short-term changes.
      • robust: Uses median and MAD (median absolute deviation). Robust to outliers; ideal for spiky, non-normally distributed metrics or detecting long-term changes.
    • anomaly_type: Defines granular per-type thresholds. Supported values:
      • requests: Request rate for a service.
      • latency: Latency for a service (e.g., p95).
      • errors: Error rate for a service.
      • resource: A gauge representing resource usage (e.g., CPU or memory).

  4. Implement a selection mechanism to avoid processing cycles

    main

    To ensure the anomaly detection framework only processes metrics intended for a specific strategy and to avoid infinite processing loops, you must define a selection rule.

    This rule should look for metrics that have an anomaly_name, an empty anomaly_select label, and the correct anomaly_strategy. It then applies a new label anomaly_select: "1". Subsequent recording rules in your strategy should ensure they carry both the anomaly_strategy and the anomaly_select labels.

    - record: anomaly:<my_strategy>:select
    expr: |- 
        {anomaly_name!="", anomaly_select="", anomaly_strategy="<my_strategy>"}
    labels:
        anomaly_select: "1"

  5. Integrate the framework with Prometheus

    main

    To use the framework, you must make the provided rules accessible to your Prometheus instance and update your Prometheus configuration to include them.

    Add the rules directory to your rule_files configuration. For example:

    rule_files:
- /etc/prometheus/rules/*.yml

  6. Run the anomaly detection demo

    main

    To see the framework in action, you can run a local demo using Docker. This will start Prometheus, Grafana, a node exporter, and an OTEL demo.

    1. Navigate to the demo directory and run make start.
    2. Access the Grafana UI at http://localhost:8080/grafana.
    3. Locate the "Anomalies" dashboard within the "Anomalies" folder.
    4. To simulate anomalies, use the load generator at http://localhost:8080/loadgen/.

    Note: Most strategies require 24-26 hours of data to be fully trained. Anomaly bands may be overly sensitive until this data threshold is met.

    cd demo
make start

  8. Implement an anomaly detection strategy using recording rules

    main

    To implement a new anomaly detection strategy, create a YAML file where you define recording and alerting rules. Every strategy must expose exactly three specific metrics to be compatible with the framework:

    1. anomaly:upper_band: The upper limit of the anomaly band.
    2. anomaly:lower_band: The lower limit of the anomaly band.
    3. anomaly:level: The baseline line used for alerts (triggered when this value falls outside the bands for a specific duration).

    To prevent infinite processing cycles, you must implement a selection mechanism using an anomaly_select label. This ensures the framework only processes metrics tagged with your specific strategy and avoids re-processing derived metrics.

    - record: anomaly:<my_strategy>:select
    expr: |- 
        {anomaly_name!="", anomaly_select="", anomaly_strategy="<my_strategy>"}
    labels:
        anomaly_select: "1"

  9. Configure the demo environment via environment variables

    main

    The docker-compose.yml file for the demo relies on several environment variables to define images, ports, and host configurations. Before running the demo, ensure the following variables are set in your environment or a .env file:

    Image and Versioning

    • IMAGE_NAME: Base name for the demo service images.
    • DEMO_VERSION: Version suffix for the demo services.
    • IMAGE_VERSION: Specific version for service images.
    • FLAGD_IMAGE: Image for the flagd service.
    • VALKEY_IMAGE: Image for the valkey-cart service.
    • GRAFANA_IMAGE: Image for the grafana service.
    • COLLECTOR_CONTRIB_IMAGE: Image for the otelcol service.
    • PROMETHEUS_IMAGE: Image for the prometheus service.
    • NODE_EXPORTER_IMAGE: Image for the node_exporter service.

    Port Mappings

    • AD_SERVICE_PORT, CART_SERVICE_PORT, CHECKOUT_SERVICE_PORT, CURRENCY_SERVICE_PORT, EMAIL_SERVICE_PORT, FRONTEND_PORT, IMAGE_PROVIDER_PORT, LOCUST_WEB_PORT, PAYMENT_SERVICE_PORT, PRODUCT_CATALOG_SERVICE_PORT, QUOTE_SERVICE_PORT, RECOMMENDATION_SERVICE_PORT, SHIPPING_SERVICE_PORT, ENVOY_PORT, GRAFANA_SERVICE_PORT, PROMETHEUS_SERVICE_PORT, NODE_EXPORTER_SERVICE_PORT.

    Telemetry and Networking

    • OTEL_COLLECTOR_HOST: Hostname/IP of the OpenTelemetry Collector.
    • OTEL_COLLECTOR_PORT_HTTP: HTTP port for the OTLP collector.
    • OTEL_COLLECTOR_PORT_GRPC: gRPC port for the OTLP collector.
    • DOCKER_SOCK: Path to the Docker socket (e.g., /var/run/docker.sock).
    • HOST_FILESYSTEM: Path to the host filesystem for the collector.
    • OTEL_COLLECTOR_CONFIG: Path to the primary OTLP collector configuration file.
    • OTEL_COLLECTOR_CONFIG_EXTRAS: Path to additional OTLP collector configuration files.

  10. Configure the OpenTelemetry Collector (otelcol)

    main

    The otelcol service acts as the central telemetry hub. It is configured via two YAML files and requires access to the host filesystem and Docker socket.

    • Commands: Uses --config=/etc/otelcol-config.yml and --config=/etc/otelcol-config-extras.yml.
    • Volumes:
      • ${HOST_FILESYSTEM} -> /hostfs:ro
      • ${DOCKER_SOCK} -> /var/run/docker.sock:ro
      • ${OTEL_COLLECTOR_CONFIG} -> /etc/otelcol-config.yml
      • ${OTEL_COLLECTOR_CONFIG_EXTRAS} -> /etc/otelcol-config-extras.yml
    • Ports: Exposes both gRPC (${OTEL_COLLECTOR_PORT_GRPC}) and HTTP (${OTEL_COLLECTOR_PORT_HTTP}) ports.
      otelcol:
    image: ${COLLECTOR_CONTRIB_IMAGE}
    container_name: otel-col
    deploy:
      resources:
        limits:
          memory: 200M
    restart: unless-stopped
    command: [ "--config=/etc/otelcol-config.yml", "--config=/etc/otelcol-config-extras.yml" ]
    user: 0:0
    volumes:
      - ${HOST_FILESYSTEM}:/hostfs:ro
      - ${DOCKER_SOCK}:/var/run/docker.sock:ro
      - ${OTEL_COLLECTOR_CONFIG}:/etc/otelcol-config.yml
      - ${OTEL_COLLECTOR_CONFIG_EXTRAS}:/etc/otelcol-config-extras.yml
    ports:
      - "${OTEL_COLLECTOR_PORT_GRPC}"
      - "${OTEL_COLLECTOR_PORT_HTTP}"
    environment:
      - ENVOY_PORT
      - HOST_FILESYSTEM
      - OTEL_COLLECTOR_HOST
      - OTEL_COLLECTOR_PORT_GRPC
      - OTEL_COLLECTOR_PORT_HTTP
    logging: *logging

  11. Configure Prometheus settings in the demo

    main

    The Prometheus service in the demo is configured with specific flags for storage, lifecycle, and OTLP support. Key configurations include:

    • --storage.tsdb.retention.time=15d: Sets data retention to 15 days.
    • --web.enable-lifecycle: Enables the API to reload configuration.
    • --web.route-prefix=/prometheus/: Sets the web route prefix.
    • --web.external-url=/prometheus/: Sets the external URL.
    • --enable-feature=exemplar-storage: Enables exemplar storage.
    • --enable-feature=otlp-write-receiver: Enables the OTLP write receiver to allow Prometheus to receive OTLP data directly.

    Volumes are used to mount the configuration and rules:

    • ./src/prometheus/prometheus-config.yaml -> /etc/prometheus/prometheus-config.yaml
    • ../rules -> /etc/prometheus/rules
      prometheus:
    image: ${PROMETHEUS_IMAGE}
    container_name: prometheus
    command:
      - --web.console.templates=/etc/prometheus/consoles
      - --web.console.libraries=/etc/prometheus/console_libraries
      - --storage.tsdb.retention.time=15d
      - --config.file=/etc/prometheus/prometheus-config.yaml
      - --storage.tsdb.path=/prometheus
      - --web.enable-lifecycle
      - --web.route-prefix=/prometheus/
      - --web.external-url=/prometheus/
      - --enable-feature=exemplar-storage
      - --enable-feature=otlp-write-receiver
    volumes:
      - ./src/prometheus/prometheus-config.yaml:/etc/prometheus/prometheus-config.yaml
      - ../rules:/etc/prometheus/rules

  12. Configure Grafana in the demo

    main

    The Grafana service is configured to automatically install the OpenSearch datasource plugin and uses custom configuration and provisioning files.

    • Plugin Installation: Uses GF_INSTALL_PLUGINS=grafana-opensearch-datasource.
    • Configuration Volume: ./src/grafana/grafana.ini is mounted to /etc/grafana/grafana.ini.
    • Provisioning Volume: ./src/grafana/provisioning/ is mounted to /etc/grafana/provisioning/ to manage dashboards and data sources.
      grafana:
    image: ${GRAFANA_IMAGE}
    container_name: grafana
    deploy:
      resources:
        limits:
          memory: 100M
    restart: unless-stopped
    environment:
      - "GF_INSTALL_PLUGINS=grafana-opensearch-datasource"
    volumes:
      - ./src/grafana/grafana.ini:/etc/grafana/grafana.ini
      - ./src/grafana/provisioning/:/etc/grafana/provisioning/
    ports:
      - "${GRAFANA_SERVICE_PORT}"
    logging: *logging