sloth

repository·main·Indexed 25 days ago

https://github.com/slok/sloth

A tool for generating uniform and reliable Prometheus Service Level Objectives (SLOs). Sloth automates the creation of SLI recording rules and complex multi-window multi-burn rate alerts based on a simple specification. It includes a variety of plugins for denominator correction, error budget exhaustion alerts, label management, rule interval configuration, and Victoria MetricsQL validation.

Tokens
17.5K
Snippets
43
Records
88
Agent score
81%

What's inside sloth

  1. Overview of Sloth features

    main

    Sloth is a tool designed to generate reliable Prometheus SLOs based on Google's SLO implementation and the multi-window multi-burn rate alert framework.

    Core capabilities include:

    • Automated Rule Generation: Creates Prometheus SLI recording rules, SLO metadata rules, and multi-window multi-burn alert rules (Page and Warning).
    • Validation: Provides a validate command for use in GitOps and CI pipelines to ensure spec correctness.
    • Flexibility: Supports custom SLI types and plugins, customizable labels, and adjustable SLO period windows (defaults to 30 and 28 days).
    • Deployment Options: Available as a single binary CLI or as a Kubernetes Controller/operator using CRDs (compatible with prometheus-operator).
    • Observability: Supports OpenSLO and provides an automatic Grafana dashboard for visualizing SLO states.
  2. Use the remove_labels_v1 SLO plugin

    main

    The remove_labels_v1 plugin removes custom labels from all SLI and metadata metrics to prevent breaking optimized 30-day recording rules.

    By default, it preserves:

    • The sloth_slo_info metric (no labels are removed from this metric).
    • SLO ID labels: sloth_service, sloth_slo, and sloth_id.
    • The sloth_window label on SLI metrics.

    You can customize this behavior using the preserveLabels and skipMetrics configuration options.

    Important: Order Requirement This plugin must run after SLI and metadata rule generation plugins.

    sloth generate --plugins-path=/plugins --slo-plugins='{"id": "sloth.dev/contrib/remove_labels/v1", "priority": 100, "config": {"preserveLabels": ["namespace"]}}' ...
  3. Use the sloth.dev/core/debug/v1 plugin for debugging

    main

    The sloth.dev/core/debug/v1 plugin is a utility for testing and debugging SLO (Service Level Objective) mutations within a plugin chain. It allows you to inspect the state of objects as they pass through the chain.

    Important: This plugin uses the debug logging level. To see its output, you must run Sloth in debug mode.

  4. What is the sloth.dev/core/sli_rules/v1 plugin?

    main

    The sloth.dev/core/sli_rules/v1 plugin is a core component of Sloth that generates Prometheus SLI error ratio recording rules for each required time window defined in an SLO. These rules serve as the foundation for other Sloth plugins, such as alerting and metadata plugins.

    It supports two types of SLIs:

    1. Event-based SLIs
    2. Raw query-based SLIs

    By default, this plugin is executed automatically by Sloth unless you define a custom plugin chain.

  5. Configure SLI types

    main

    An SLI (Service Level Indicator) defines what constitutes 'good' or 'bad' for an SLO. Sloth requires queries to use the {{.window}} template variable to calculate indicators over time windows.

    You must choose exactly one of the following SLI types:

    1. SLIEvents: Calculates a ratio by dividing bad events by total events. Requires an errorQuery (e.g., http 5xx) and a totalQuery (e.g., all http requests).
    2. SLIRaw: Used when the error ratio (0-1) is already calculated by an external system or recording rule. Requires an errorRatioQuery.
    3. SLIPlugin: Uses a pluggable mechanism. Requires an id (plugin name) and optional options map.
    type SLI struct {
        // Raw is the raw SLI type.
        // +optional
        Raw *SLIRaw `json:"raw,omitempty"`
    
        // Events is the events SLI type.
        // +optional
        Events *SLIEvents `json:"events,omitempty"`
    
        // Plugin is the pluggable SLI type.
        // +optional
        Plugin *SLIPlugin `json:"plugin,omitempty"`
    }
  6. Define Service Level Indicators (SLI)

    main

    An SLI (Service Level Indicator) defines how to measure the health of an SLO. Sloth supports three types of SLIs. You must use exactly one of the following types in your configuration:

    1. Events (SLIEvents): Calculates a ratio by dividing bad events by total events. Requires error_query and total_query Prometheus queries using the {{.window}} template variable.
    2. Raw (SLIRaw): Uses a pre-calculated error ratio (0-1) from an existing Prometheus query via error_ratio_query.
    3. Plugin (SLIPlugin): Uses a custom plugin to determine the SLI, identified by an id and provided via options.
  7. Configure SLO plugins and plugin chains

    main

    Sloth uses a plugin system to modify the SLO generation process. You can define a chain of SLOPlugin objects.

    Plugin Priority

    Priority determines the execution order in the chain.

    • Lower numbers have higher priority (executed earlier).
    • Default plugins use priority 0.
    • To execute plugins before the defaults, use negative priority (e.g., -100).
    • To execute plugins after the defaults, use positive priority (e.g., 100).
    • It is recommended to use large gaps between priority numbers (e.g., 10, 100, 1000, -200, -1000) to allow for future insertions.

    Overriding Plugins

    Using SLOPlugins, you can control how plugins are inherited:

    • If OverridePrevious is set to true, it overrides the plugins declared at higher levels.
    • The declaration order is: default plugins -> SLO Group plugins -> SLO plugins.
    type SLOPlugin struct {
        // ID is the ID of the plugin to load .
        ID  string `json:"id"` 
    
        // Config is the configuration used on the plugin instance creation.
        Config json.RawMessage `json:"config,omitempty"` 
    
        // Priority is the priority of the plugin in the chain. The lower the number
        // the higher the priority. The first plugin will be the one with the lowest
        // priority. The default plugins loaded by Sloth use `0` priority. If you want to
        // execute plugins before the default ones, you can use negative priority.
        // It is recommended to use round gaps of numbers like 10, 100, 1000, -200, -1000...
        Priority int `json:"priority,omitempty"` 
    }
    
    type SLOPlugins struct {
        // OverridePrevious will override the previous SLO plugins declared.
        // Depending on where this SLO plugins block declared will override:
        // - If declared at SLO group level: Overrides the default plugins.
        // - If declared at SLO level: Overrides the default + SLO group plugins.
        // The declaration order is default plugins -> SLO Group plugins -> SLO plugins.
        OverridePrevious bool `json:"overridePrevious,omitempty"` 
    
        // Chain is the list of plugin chain to add to the SLO generation.
        Chain []SLOPlugin `json:"chain"` 
    }
  8. Configure SLO Plugins

    main

    Sloth uses a plugin chain to process SLOs during rule generation. You can manage these via SLOPlugins:

    • OverridePrevious: If true, this block overrides previously declared plugins. The declaration order is: Default Plugins -> SLO Group Plugins -> SLO Plugins.
    • Chain: A list of SLOPlugin objects.

    Each SLOPlugin contains:

    • ID: The plugin identifier.
    • Config: A json.RawMessage for plugin-specific configuration.
    • Priority: Determines execution order. Lower numbers have higher priority. Default plugins use priority 0. To run before defaults, use negative numbers (e.g., -100). To run after, use positive numbers. It is recommended to use large gaps (e.g., 10, 100, 1000) for priority values.
  9. Configure SLO Alerting

    main

    The Alerting configuration manages how Sloth generates Prometheus alerts for an SLO. It includes:

    • Name: The base name for generated alerts.
    • Labels: Global Prometheus labels applied to all alerts from this SLO.
    • Annotations: Global Prometheus annotations applied to all alerts from this SLO.
    • PageAlert: Configuration for critical alerts (e.g., multi-window multi-burn rate alerts).
    • TicketAlert: Configuration for warning alerts.

    Each specific alert (PageAlert or TicketAlert) uses the Alert type, which allows you to:

    • Disable: Set to true to prevent Sloth from generating this specific alert.
    • Labels: Override or add specific labels for that alert type.
    • Annotations: Override or add specific annotations for that alert type.
  10. How the sloth.dev/core/validate/v1 plugin works

    main

    The sloth.dev/core/validate/v1 plugin is a safety check that validates the SLO specification against the Prometheus SLO dialect. It ensures the specification is correct and well-formed before any rules are generated or other plugins are executed.

    This plugin is enabled by default. You should only disable it if you are using a custom backend (such as VictoriaMetrics or Loki) that requires different validation logic. If you disable it, you should replace it with a custom validator plugin tailored to your specific target system.

  11. Use the no-op plugin as a placeholder or test tool

    main

    The sloth.dev/core/noop/v1 plugin performs no operation. It is intended to be used as a placeholder in a plugin chain or as a minimal reference implementation when building new SLO plugins. It can also be used to verify that the plugin chain mechanism is functioning correctly.

    chain:
      - id: "sloth.dev/core/noop/v1"