Rust Telemetry Workshop

Prometheus is a widely adopted time-series database for production environments. It is frequently used in conjunction with:

• Grafana: A dashboarding tool used to explore and visualize the metric data collected by Prometheus.
• Alertmanager: A system used to trigger alerts based on specific metric queries (e.g., triggering an alert if the percentage of failed requests exceeds 5% for more than 15 minutes).

When exporting metrics, you can choose between two primary architectural patterns:

1. Push-based model: The application is responsible for sending metric data to a storage system (e.g., over the network or to a file). The application controls the exporting schedule.
2. Pull-based model: The application is passive and exposes an endpoint (e.g., an HTTP endpoint) containing the current values of all recorded metrics. An external system (like Prometheus) periodically "scrapes" this endpoint to collect data.

Prometheus is most commonly used in a pull-based configuration, typically with a scrape interval of around 15 seconds.

To operate highly available services, you must define measurable dimensions that correlate with user experience. These are categorized into three concepts:

• Service Level Indicators (SLIs): The actual measurable dimensions (e.g., payment success rate, settlement time, or notification latency).
• Service Level Objectives (SLOs): Self-imposed targets for your SLIs.
• Service Level Agreements (SLAs): Targets that are part of a formal contract with your users. Typically, SLOs are set to be stricter than SLAs.

When these targets are breached, it should trigger an alert for operator engagement.

In Rust, errors are expected to implement the std::error::Error trait. This trait requires the implementation of two other traits: Debug and Display. Each serves a different purpose for telemetry and troubleshooting:

• Debug: Intended for operators. It exposes internal details of the error and the system. You can typically implement this using #[derive(Debug)].
• Display: Intended for users. It provides a human-readable description that does not expose system internals.
• source(): Allows you to walk the chain of errors. Since errors often wrap lower-level errors (e.g., a database error wrapping a network error), the source method enables inspection of each semantic layer in the failure chain.

use std::fmt::{Debug, Display};

pub trait Error: Debug + Display {
    fn source(&self) -> Option<&(dyn Error + 'static)>;
}

In Rust, panics are used for (almost) unrecoverable failures that unwind the stack. You should trigger a panic in the following scenarios:

• You encounter a situation that cannot be recovered from.
• Continuing execution would be unsafe.
• The situation does not warrant sophisticated error handling (e.g., missing a critical resource required for the application to function).
• An internal application invariant has been violated.
• In testing code, when a specific condition must be met for the test to proceed.

When choosing a metric type, use Gauges instead of Counters if the value you are measuring can increase or decrease over time.

• Counters: Must be positive and monotonically increasing (only go up). Use these for cumulative totals like total_requests_received.
• Gauges: Represent a snapshot of a current value. Use these for properties that fluctuate, such as concurrent requests, idle database connections, or available disk space.

Like counters, gauges support metadata such as descriptions, units, and labels.

A key constraint of tracing is that fields must be declared when the span is created. If you attempt to .record a key that was not part of the initial span definition, the call will be silently ignored at runtime.

To handle values that are only determined during execution, define the field upfront using tracing::field::Empty as its initial value. This allows you to use .record later to populate the field without incurring the overhead of dynamic field discovery.

// Define the field as Empty upfront so it can be recorded later
let span = tracing::info_span!("my special task", foo = tracing::field::Empty);

// Now you can successfully record the value
span.record("foo", 43);

Implementing the Subscriber trait directly is complex because it requires managing low-level details like unique span ID generation and hierarchical linking. Instead, you should use the following pattern:

1. Use Registry as a foundation: The Registry handles the complicated management of span IDs and the internal state of the tracing tree.
2. Implement or use Layer: Instead of implementing Subscriber, you implement the Layer trait. A Layer is a modular component that can be plugged into a Registry to perform specific tasks (like logging to a file or sending data to a collector).
3. Combine layers: You can stack multiple layers together using the .with() method (provided by the SubscriberExt trait) to build a complex subscriber from simple, reusable parts.

When deciding what constitutes a 'unit of work', avoid modeling every single function call as a unit, as this creates excessive noise. Instead, select units that are meaningful and occupy a significant amount of time relative to the overarching operation (e.g., an incoming web request).

Examples for a web server:

• Bad candidate: Parsing a header value (too fast, low variance in duration).
• Good candidate: Parsing the request body (takes significant time, duration varies based on body size and network conditions).

Always evaluate granularity based on the context of the overarching operation you are monitoring.

The metrics crate follows the facade pattern, similar to log and tracing. It provides a set of macros for creating and manipulating metrics without being tied to a specific storage or processing backend.

To actually process the metric data, you must provide an implementation of the metrics::Recorder trait. The facade macros forward all metric data to this Recorder implementation, which handles the actual processing, aggregation, or exporting of the data.