DOCSEARCHbyAlgolia
Submit Docs

StatsHouse Monitoring System

repository·master·Indexed 43 days ago

https://github.com/vkcom/statshouse

A high-scale, multitenant monitoring system designed for massive metric ingestion rates with low latency and automatic data downsampling. Includes tools for V2 to V3 data migration, a Grafana Data Source Backend Plugin template, and specialized Go implementations for SQLite with dual-WAL support and ClickHouse logging.

Tokens
60.2K
Snippets
129
Records
295
Agent score
54%

What's inside StatsHouse

  1. Overview of StatsHouse monitoring system

    master

    StatsHouse is a highly available, scalable, and multitenant monitoring system designed for large-scale production environments. It is optimized for high-resolution (1-second default) and low-latency (5-second default) metric observation.

    Key Features

    • Scalability: Capable of handling billions of metrics per second.
    • Automatic Downsampling: High-resolution data is automatically downsampled to 1-minute and 1-hour resolutions with automatic TTL:
      • High-resolution (1s): Stored for 2 days.
      • Minute-resolution (1m): Stored for 1 month.
      • Hour-resolution (1h): Stored indefinitely.
    • Resilience: Automatically handles network unavailability and machine failures. Most components are quasi-stateless, utilizing a ClickHouse cluster as the primary stateful data store.
    • Load Management: Uses explicitly configured per-second budgets for network and disk usage, automatically sampling metrics that exceed their budget to prevent system overload.
    • Visualization: Includes a built-in UI for interactive metric exploration and dashboards, as well as a Grafana data source plugin for advanced use cases.

  2. Overview of StatsHouse features

    master

    StatsHouse is a highly available, scalable, and multitenant monitoring system designed for large-scale production environments. Key capabilities include:

    • High Availability & Scalability: Delivers uninterrupted service during failures and handles growing data volumes while providing fair resource sharing among multiple user groups.
    • High-Resolution Data: Provides near real-time monitoring with a default one-second resolution and five-second latency.
    • Long-term Storage via Downsampling: Automatically manages data retention by downsampling metrics:
      • Per-second aggregated data: Stored for 2 days.
      • Per-minute aggregated data: Stored for 1 month.
      • Per-hour aggregated data: Stored indefinitely.
    • Operational Resilience: Designed to handle network unavailability, machine failures, and sharp increases in data insertion or client-side errors. Most components are quasi-stateless, relying on a ClickHouse cluster as the primary stateful data store.
    • Overload Resistance: Uses explicitly configured per-second budgets for network and disk usage. To ensure fair resource sharing, StatsHouse automatically samples metric data if a specific metric exceeds its allocated budget.
    • Built-in Visualization: Offers an interactive user interface that does not require a query language. Users can utilize the built-in dashboard system or the Grafana data source plugin.

  3. Introduction to StatsHouse

    master

    StatsHouse is an open-source, highly available, scalable, and multitenant monitoring system. It is designed to handle massive amounts of metric data from applications without being overloaded, utilizing aggregation and sampling to manage load.

    Users can send metric data without limitations and visualize statistics in real-time via a graph-based user interface.

  4. Use the StatsHouse API and UI

    master

    StatsHouse provides a thin API client and a User Interface for interacting with metric data.

    • API: The API client allows for efficient queries while minimizing direct load on the ClickHouse cluster. The service implements caching to protect the database from ineffective queries. For the full specification, refer to the OpenAPI documentation.
    • UI: The User Interface retrieves data via the StatsHouse API and renders metric data in a graph view.

  5. What is cardinality and how does it affect performance?

    master

    Cardinality is the number of unique tag value combinations sent for a specific metric.

    Key points for developers:

    • Data Volume: The amount of inserted data depends on cardinality, not the total number of events. StatsHouse collapses rows with identical tag combinations and summarizes their counters.
    • Database Impact: When querying data, the system must iterate over rows for the chosen time interval. Higher cardinality results in more rows, which increases the time required to retrieve and process the data.
    • Aggregation Impact: Cardinality can increase during between-host aggregation if different hosts report different sets of tag combinations.

  6. How PromQL works in StatsHouse

    master

    Unlike Prometheus, which stores timestamp—value pairs, StatsHouse stores aggregates per time interval. This means query results are aggregates rather than exact metric values at a specific moment.

    An aggregate contains several components:

    • count
    • sum
    • min
    • max
    • tag_s (String top tag, if enabled)
    • percentiles (if enabled)

    The specific aggregate returned depends on the minimal available aggregation interval, the requested aggregation interval, and the metric resolution.

  7. Compare New and Old Dataflow Pipelines

    master

    The dataflow architecture has evolved from a dual-pipeline (old tables) to a unified-capable pipeline (new tables) that can run in parallel during migration.

    New Dataflow

    Aggregates flow into the input_table, which then branches into both the new tables and old tables pipelines simultaneously to support migration.

    New Tables Path: input_table $\rightarrow$ new_map_view_1s/m/h/h_short $\rightarrow$ new_table_1s/m/h/h_short

    Old Tables Path: input_table $\rightarrow$ map_view_1s/m/h & prekey_map_view_1s/m/h $\rightarrow$ table_1s/m/h & prekey_table_1s/m/h

  8. Use tags to differentiate and filter metrics

    master

    Tags are name-value pairs used as additional dimensions to filter or group your data (e.g., differentiating by platform, region, or environment). In the StatsHouse UI, you can use these tags to answer questions like "Does the error rate differ for platforms?" or "What is the region with the highest request rate?".

    When sending metrics via a client library, you provide a dictionary of tags where keys are tag names and values are tag values.

    # Example: sending a value metric with multiple tags
statshouse.value("request_rate", {"env": "production", "region": "moscow"}, 42.5)

  9. Understand the Metadata component and string-to-int32 mapping

    master

    The Metadata component manages a global string ↔ int32 map used to optimize performance and reduce latency. This map translates metric names and tag values (strings) into int32 integers.

    Key characteristics:

    • Commonality: The map is shared across all metrics.
    • Persistence: Elements in this map are never deleted.
    • Optimization: Mapping strings to integers allows StatsHouse to provide real-time data with low latency.

    Example mapping:

    • 'iphone' ↔ 12
    • 'null' ↔ 26

  10. Understand aggregation intervals and resolution

    master

    The aggregation interval determines the resolution of your metric data. A larger interval results in a smoother graph.

    Auto Interval Behavior: When set to Auto, StatsHouse uses the minimal available interval for aggregation, which varies based on data age:

    • Per-second data: Stored for the first two days.
    • Per-minute data: Stored for one month.
    • Per-hour data: Stored indefinitely.

  11. How StatsHouse sampling works

    master

    StatsHouse uses sampling to prevent data lag caused by bottlenecks in data transmission (agent to aggregator) or database insertion. When the volume of incoming data exceeds the system's capacity, the system applies a sampling factor to discard "extra" data while scaling the remaining values to preserve the expected value (mathematical expectation) of the metrics.

    Key Characteristics:

    • Preservation of Averages: Sampling preserves average values across aggregates (e.g., 1-minute or 1-hour windows), though it introduces high-frequency noise.
    • Impact of Cardinality: Higher metric cardinality leads to higher sampling factors, which increases noise on graphs. This serves as a system incentive to keep cardinality low.
    • Data Types: Because sampling factors can be non-integers (e.g., 1.5), counters in StatsHouse are stored as floating-point numbers rather than integers.
    • Rounding Option: You can configure specific metrics to use random rounding for sampling factors. For example, a factor of 1.1 would result in a value of 1 nine times out of ten, and 2 once out of ten.
    • Hierarchical Application: Sampling is applied hierarchically: first across Business Units, then across Groups within a unit, and finally across Metrics within a group.

  12. Understand StatsHouse data resolution and aggregation

    master

    Unlike Prometheus, which stores raw time-value pairs, StatsHouse stores aggregated values (a "digest") for specific intervals. This means you cannot retrieve an exact value for an arbitrary moment in time, as only the aggregates are preserved.

    Data resolution changes based on the age of the data:

    • 0 to 2 days old: Stored as per-second aggregates.
    • 2 to 33 days old: Stored as per-minute aggregates.
    • 33+ days old: Stored as per-hour aggregates.

    When querying, StatsHouse calculates a time grid by splitting the requested interval at resolution boundaries (2 and 31 days) and determining a step size based on the interval's minimum step, the metric's resolution, and the requested step (rounded up to the nearest 1, 5, 15, or 60 second/minute increment).