DOCSEARCHbyAlgolia
Submit Docs

pint

repository·main·Indexed 21 days ago

https://github.com/cloudflare/pint

A specialized linter and validator for Prometheus rules designed to ensure rule files follow best practices and are syntactically correct. It provides a CLI with commands for linting, parsing PromQL queries, and continuous monitoring via a watch mode that exposes metrics via HTTP. Pint supports both online checks requiring Prometheus server connectivity and offline structural checks.

Tokens
35K
Snippets
132
Records
181
Agent score
72%

What's inside pint

  2. What is the alerts/comparison check?

    main

    The alerts/comparison check enforces the use of comparison operators (e.g., > 10 or > 0) in alert queries.

    Without a comparison operator, an alerting rule query returns the metric series itself. Since any returned result triggers an alert, a query like errors will trigger an alert even when the value is 0. Using a condition like errors > 10 ensures alerts only trigger when the specific threshold is met.

    While some metrics (like http_responses_total{code="504"}) might only exist in Prometheus when they have non-zero values, relying on this behavior is fragile. It is a best practice to always include a comparison operator to ensure predictable alerting behavior.

  3. Understand the promql/impossible check

    main

    The promql/impossible check identifies PromQL queries that are logically guaranteed to return no results. This typically happens when operators or aggregations strip away labels required for matching, joining, or grouping.

    Common scenarios detected include:

    1. Label Mismatches in Set Operators: Using unless, and, or or where the right-hand side has a different label set than the left-hand side (e.g., using sum() on the right side removes all labels, making it impossible to match a left-hand side that has labels).
    2. Aggregating on Missing Labels: Using group by on labels that were already removed by an inner aggregation (e.g., group by(cluster) (sum(errors)) fails because sum() removes the cluster label).
    3. Joins on Missing Labels: Performing a join on(label) where the label has been stripped by an aggregation on one or both sides.
    4. Binary Aggregations blocking label propagation: Using group_left or group_right in a way that prevents labels from propagating to the final result when combined with other operators (e.g., missing parentheses around a complex expression).
    // Example of an impossible query: labels on LHS won't match empty labels on RHS
foo{job="bar"} unless sum(foo)

// Example of aggregating on a label that was already removed
group by(cluster) (
  sum(errors)
)

// Example of a join on a label that was already removed
sum(foo) / on(cluster) count(bar)

  4. Assign owners to rules using control comments

    main

    You can use special comments to assign owners to rules, which allows you to route alerts based on the owner label in the pint_problem metric.

    Requirements:

    • Comments must start with # pint (with a space).
    • Use # pint file/owner <owner> to set an owner for an entire file.
    • Use # pint rule/owner <owner> to set an owner for a specific rule.

    Example:

    # pint file/owner bob

- alert: AlertName
  expr: up == 0

# pint rule/owner alice
- alert: OtherAlert
  expr: up == 1

  5. What the `promql/range_query` check does

    main

    The promql/range_query check inspects range query selectors (e.g., foo[40d]) in PromQL queries. It warns if a query requests a time range that exceeds the configured Prometheus retention limits.

    By default, Pint checks the --storage.tsdb.retention.time flag passed to Prometheus. If Prometheus is configured with --storage.tsdb.retention.time=30d, a query like foo[40d] will trigger a warning because the query can only ever return up to 30 days of data, potentially leading to a mismatch between user expectations and actual results.

  6. Understand the `rule/dependency` check

    main

    The rule/dependency check validates Prometheus rule dependencies to prevent broken metrics or data inconsistency. It covers two main scenarios:

    1. Removed dependencies: When running pint ci, it ensures that if you delete a recording rule, it isn't still being used as a dependency by other rules. Deleting a dependency without updating its consumers will cause those consumers to fail.
    2. Cross-group dependencies: It warns when a recording rule in one group uses a metric produced by a recording rule in a different group. Because groups run at different times, the dependent rule will see stale data (up to one evaluation interval of lag), causing metric mismatches.

    Best Practice: To avoid lag and ensure consistency, move dependent recording rules into the same group so they execute sequentially within the same evaluation cycle.

  7. Understand the alerts/external_labels check

    main

    The alerts/external_labels check ensures that Prometheus alerting rules referencing $externalLabels variables actually correspond to labels configured in the Prometheus server's global:external_labels section.

    If an alert rule uses a variable like {{ $externalLabels.cluster }} but cluster is not defined in the Prometheus server's global:external_labels configuration, this check will report an error because the variable will resolve to an empty value.

    # Prometheus configuration
global:
  external_labels:
    cluster: mycluster

# Alert rule using the label
- alert: Abc Is Down
  expr: up{job="abc"} == 0
  annotations:
    summary: "{{ $labels.job }} is down in {{ $externalLabels.cluster }} cluster"

  8. What the promql/features check does

    main

    The promql/features check ensures that PromQL queries do not use experimental features that require specific --enable-feature=... flags which are not currently enabled on the target Prometheus server.

    To perform this validation, Pint:

    1. Queries the Prometheus flags API to retrieve enabled feature flags.
    2. Queries the build info API to check the Prometheus version against feature requirements.

    Currently monitored features include:

    • promql-experimental-functions: Required for functions like mad_over_time, sort_by_label, sort_by_label_desc, info, double_exponential_smoothing, min_of, max_of, start, end, range, step, limitk, and limit_ratio.
    • promql-duration-expr: Required for arithmetic expressions in time durations.
    • promql-extended-range-selectors: Required for anchored or smoothed range selector modifiers.
    • promql-binop-fill-modifiers: Required for the fill() binary operator modifier.

  9. Understand the promql/counter check

    main

    The promql/counter check identifies Prometheus alerting rules that use counter metrics incorrectly.

    The Concept: Counters track the number of events over time and can only grow (until they overflow or the service restarts and resets to zero). Because the absolute value of a counter is essentially a random number depending on the application's uptime, using a raw counter in an expression like errors_total > 10 is usually invalid for alerting. Such an alert would fire once the threshold is hit and stay firing until a restart, regardless of whether errors are still occurring.

    The Correct Pattern: To track the health of an application in real-time, you should use counter-safe functions like rate() to calculate the frequency of events over a specific time window. This ensures that if the error rate drops, the alert stops firing.

    Example of an invalid rule:

    - alert: Too many errors
  expr: errors_total > 10

    Example of a valid rule:

    - alert: Too many errors
  expr: rate(errors_total[1h]) > 10

  10. Ignored cases for the promql/series check

    main

    The promql/series check ignores certain query patterns to avoid false positives:

    vector() fallback

    Metrics wrapped in ... or vector(0) are ignored because the or vector(0) pattern is explicitly intended to provide a fallback value when no time series match.

    - alert: Foo
  expr: sum(my_metric or vector(0)) > 1

    foo unless bar

    In a query using foo unless bar, only the foo selector is tested. The check assumes the presence of bar is not guaranteed, so it only verifies if bar exists rather than checking if it matches the query's intent.

    Note: If the query is foo unless bar > 5, both foo and bar will be tested because the presence of a value comparison (> 5) implies the metric is assumed to be present.

  12. What the promql/rate check inspects

    main

    The promql/rate check analyzes rate() and irate() function calls in PromQL queries to ensure they are used correctly. It validates three main criteria:

    1. Valid Time Durations: It ensures range queries use a duration of at least 2x the global scrape_interval of the selected Prometheus servers. This is necessary because Prometheus requires at least two samples to calculate a rate.
    2. Counter Metric Usage: It verifies that metrics passed to rate() or irate() are counters. Using non-counter metrics (like gauges) can lead to incorrect results due to how counter overflows are handled. For gauge metrics, use delta() or deriv() instead.
    3. Correct Chaining: It checks that rate() is not called on the result of sum(counter), as this produces invalid results. It specifically looks for the pattern rate(sum(...)) and flags it unless the metric is produced via recording rules.

    Note on Metadata Mismatches: Metric type checks rely on the Prometheus metadata API. If a metric name is exported with multiple different types across different targets, or if a type change hasn't propagated to all targets yet, the check may report errors due to ambiguous metadata.