Gnocchi Time Series Database

Gnocchi is an open-source time series database designed for the storage and indexing of time series data and resources at a large scale. It is specifically architected for modern, dynamic, and potentially multi-tenant cloud platforms.

Key characteristics include:

• Scalability and Fault-Tolerance: Designed to handle large amounts of aggregates while remaining performant.
• Storage Agnostic: Built to avoid hard dependencies on complex storage systems.
• Pre-computed Aggregations: Unlike many time series databases that compute aggregations (like average or minimum) at query time, Gnocchi aggregates data points at ingestion. This makes data retrieval extremely fast because the system simply reads back pre-computed results.

The gnocchi-nagios service acts as a bridge between monitoring tools and Gnocchi. It performs the following workflow:

1. Collects: It gathers perfdata files generated by Nagios or Icinga.
2. Transforms: It converts the performance data into Gnocchi-compatible formats, specifically mapping them to Gnocchi |resources|, |metrics|, and |measures|.
3. Publishes: It sends the transformed data to the Gnocchi REST API.

For more detailed information, refer to the official Documentation or the Source Code.

The gnocchi-amqp1d daemon acts as an intermediary for metrics sent over the network via the amqp1 collectd plugin (compatible with brokers like Apache Artemis or Apache Qpid Dispatch Router).

Workflow:

1. Metrics are published to a specific host, port, and topic via the amqp1 plugin.
2. The gnocchi-amqp1d daemon receives these metrics.
3. Metrics are created dynamically upon receipt.
4. Each metric is attached to the configured Resource Name.
5. The source hostname is automatically attached to the resource as an attribute.

While gnocchi-statsd can be scaled, the nature of the Statsd protocol introduces specific constraints depending on the metric type used:

• Counters (counter) and Sampling (c): You must always send these metrics to the same daemon instance. If sent to different instances, the counts will not be correctly aggregated. If you need to scale, avoid using these types or ensure sticky routing to a single daemon.
• Timings (timing) and Gauges (gauges): These types do not suffer from the same scaling limitation, but be aware that sending the same metric to different gnocchi-statsd servers may result in more measures than expected because the cache and flush delays are not synchronized across instances.

To use Gnocchi effectively, you must understand the relationship between its three primary data entities:

1. Resource: The base entity representing an infrastructure component (e.g., a virtual machine or a network interface). It is identified by a unique ID and can hold metadata via attributes. You associate metrics with resources using a name.
2. Metric: An entity identified by a UUID that stores aggregated data. It is attached to a Resource and its storage behavior is governed by an Archive policy.
3. Measure: The raw data input. A measure is a tuple consisting of a timestamp and a value sent to the Gnocchi API.

In short: You send Measures to a Metric, which is attached to a Resource.

Archive policies define how metrics are aggregated and how long they are stored.

Key Concepts:

• Definition: Expressed as a number of points over a timespan (e.g., 10 points with 1s granularity = 10s retention).
• Expiration: Data is expired based on the timespan, not a relative timestamp.
• Aggregation Methods: Each policy uses specific methods (default: mean, min, max, sum, std, count).
• Performance: More definitions in a single policy increase CPU consumption.

Example Calculation: To keep 24 hours of data with 1-minute granularity: 1440 points with a granularity of 1 minute.

The Gnocchi API server supports three authentication methods:

1. basic (Default): Uses the standard HTTP Authorization header. By default, only the admin user has special permissions. The authentication password is not used.
2. keystone: Uses OpenStack Keystone. To enable this, set api.auth_mode = keystone and configure the keystone_authtoken section in gnocchi.conf with the appropriate values.
3. remoteuser: Gnocchi retrieves the username from the HTTP server's REMOTE_USER environment variable. Permissions follow the basic mode model.

Gnocchi is a stateless, horizontally scalable 'Metric as a Service' system composed of several services:

• HTTP REST API: The primary entry point for receiving data.
• statsd-compatible daemon: An optional daemon for receiving metrics via the statsd protocol.
• gnocchi-metricd: An asynchronous processing daemon that performs background operations such as statistics computing and metric cleanup.

Because the services are stateless, you can scale horizontally by spawning more gnocchi-api endpoints or gnocchi-metricd daemons to handle increased load or to ensure high availability across independent servers.

Gnocchi translates Prometheus metrics into its internal resource model using the following logic:

1. Resource Creation: For every metric sent by Prometheus, Gnocchi creates (or identifies) a resource based on the unique combination of the job and instance labels.
2. Resource Type: These resources are assigned the prometheus resource type.
3. Attributes: Each resource contains two specific attributes: job and instance.
4. Measures: The actual metric values sent by Prometheus are stored as measures attached to that specific job/instance resource.

The /v1/prometheus/write endpoint processes these incoming metrics using the WriteRequest protobuf message.