DOCSEARCHbyAlgolia
Submit Docs

Thanos

repository·main·Indexed 12 days ago

https://github.com/thanos-io/thanos

A CNCF Incubating project providing a highly available, scalable, and long-term storage solution for Prometheus metrics. It enables global querying across multiple Prometheus instances, deduplication of HA pairs, and unlimited metric retention using object storage. Key components include Sidecar and Receive deployments, a React-based UI, and the thanos-mixin for Jsonnet-based configuration of Grafana dashboards and Prometheus alerting rules.

Tokens
123.5K
Snippets
294
Records
496
Agent score
96%

What's inside Thanos

  1. Overview of Thanos Receive

    main

    The thanos receive component implements the Prometheus Remote Write API. It acts as a middle layer that receives metrics from Prometheus instances, stores them in a local TSDB, and uploads TSDB blocks to object storage (e.g., S3) every 2 hours by default.

    Key features include:

    • Horizontal Scalability: Distributes incoming series across multiple Receiver nodes.
    • Long-term Storage: Automatically uploads blocks to object storage.
    • Multi-tenancy: Supports multiple tenants using labels.
    • Exemplars Support: Ingests and stores exemplars via remote-write.
    • StoreAPI & ExemplarsAPI: Exposes APIs so Thanos Queriers can query received metrics and exemplars in real-time.

    Use Case: Recommended for air-gapped or egress-only environments where Prometheus can only push metrics.

    Important: Ensure you set correct external labels to identify data blocks across Thanos clusters.

  2. Overview of Thanos capabilities

    main

    Thanos is a set of components designed to create a highly available metric system with unlimited storage capacity. It is built to be added seamlessly on top of existing Prometheus deployments by leveraging the Prometheus 2.0 storage format.

    Key objectives include:

    1. Global query view: A unified view of metrics across all connected Prometheus servers.
    2. Unlimited retention: Cost-efficiently storing historical metric data in any object storage.
    3. High availability: Providing HA for components, including Prometheus, and merging data from Prometheus HA pairs on the fly.

  3. Thanos Query API Extensions

    main

    The Thanos Query API is compatible with the Prometheus 2.x API but includes several extensions for handling distributed data:

    • Partial Response: Allows queries to succeed even if some underlying StoreAPIs fail or timeout, returning a 200 OK with warnings instead of an error.
    • Deduplication: Uses replica labels to merge duplicate metrics from High Availability (HA) groups.
    • Auto Downsampling: Automatically selects appropriate data resolution.
    • Store Filtering: Allows targeting specific stores via matchers.
    • Custom Response Fields: Adds a warnings field to the JSON response to communicate non-critical errors.

  4. Understand Thanos core features and deployment model

    main
    Thanos provides a global query view, high availability, and data backup with historical, cheap data access. All these features are contained within a single binary, but they can be deployed independently. This modularity allows you to deploy a subset of features for immediate benefit or to perform gradual rollouts in complex environments.

  6. Use the `thanos tools` subcommand for development and debugging

    main

    The thanos tools subcommand provides a collection of short-lived CLI utilities designed for development, debugging, and administrative tasks related to Thanos data and configuration. These tools are distinct from the long-running Thanos components (like Sidecar, Store, or Compactor).

    thanos tools <command> [<args> ...]

  7. What is the Thanos Sidecar and how does it work?

    main

    The thanos sidecar is a component deployed alongside a Prometheus instance. It serves two primary functions:

    1. StoreAPI Implementation: It implements the Thanos Store API on top of Prometheus' remote-read API. This allows Thanos Queriers to query Prometheus data using the efficient StoreAPI without needing to interact with Prometheus APIs directly.
    2. Object Storage Upload (Optional): It can optionally upload TSDB blocks to an object storage bucket every 2 hours as Prometheus produces them. This enables running Prometheus with low local retention while maintaining durable, queryable historical data in object storage.

    Important Note on Statelessness: Even with sidecar uploading, Prometheus is not fully stateless. If Prometheus crashes, you may lose up to ~2 hours of metrics. Persistent disk for Prometheus is highly recommended.

  8. What is the StoreAPI and how is it used?

    main

    The StoreAPI is a common gRPC proto interface used by Thanos components (like the Querier) to fetch metric series from various storage backends.

    While Thanos natively implements the StoreAPI via its own Sidecar, Ruler, and Store gateway components to interface with Prometheus or Prometheus TSDB formats, the interface is generic. This allows third-party storage systems to integrate with Thanos by implementing this same gRPC interface, enabling a unified query layer across different data sources.

  9. How Vertical Query Sharding works via QueryAnalyzer

    main

    The sharding process relies on a QueryAnalyzer component. The QueryAnalyzer performs the following steps:

    1. AST Traversal: It traverses the PromQL Abstract Syntax Tree (AST).
    2. Label Extraction: It identifies specific labels within the query that can be used as sharding keys.
    3. Dataset Partitioning: These labels are used to partition the dataset so that individual stores return only a specific subset of the series.

    This mechanism allows the system to distribute the workload of a single PromQL query across multiple Queriers based on the hash of label value pairs.

  10. Configure Partial Response Strategy

    main

    The Partial Response behavior controls the tradeoff between accuracy and availability. When a StoreAPI returns an error or times out, Thanos can either fail the entire query or return a partial result with warnings.

    Parameters

    • partial_response (Boolean): If true, unavailable StoreAPIs will not cause the query to fail; instead, a warning is returned. Defaults to the value of the --query.partial-response flag (which defaults to true).

    Strategies

    • warn: Returns the available data along with human-readable warnings in the response.
    • abort (default): Fails the query if any StoreAPI fails.

    Timeout Configuration

    To prioritize availability, you can set a tighter timeout for underlying stores than the overall query timeout using:

    • --query.timeout
    • --store.response-timeout

  11. Thanos deployment architectures

    main

    Thanos can be deployed using two primary patterns depending on your requirements:

    1. Sidecar Deployment (for Kubernetes)

    In this model, a Thanos Sidecar is deployed alongside each Prometheus instance. This allows Thanos to access the local Prometheus storage directly.

    2. Receive Deployment (for scaling or remote write)

    This model is used to scale out or to implement integration with other remote-write compatible sources. Data is sent to a Thanos Receive component, which then manages the ingestion.

  12. How tenancy works in the Thanos query path

    main

    In the Thanos query path, tenancy is modeled based on the query initiator rather than the data owner (which is how Thanos Receive models it).

    1. Identification: An incoming HTTP request to the Query Frontend or Querier includes a tenant identifier in a configurable HTTP header (e.g., THANOS-TENANT).
    2. Propagation: The tenant ID is passed downstream to all components using a standardized internal header for HTTP and gRPC metadata for gRPC calls.
    3. Observability: Components like Query Frontend, Querier, and Store Gateway include the tenant label in their exported metrics, allowing for per-tenant monitoring.
    4. Enforcement: When enabled in the Querier, the system automatically injects a label matcher into the PromQL query to ensure the results are restricted to the identified tenant's data.