Grafana Agent

The Flow mode documentation is organized into five distinct types of content to help users navigate from onboarding to deep technical reference:

1. Get started: Onboarding content for new users, including installation guides and "hello world" configurations to help decide if Flow mode is a good fit.
2. Concepts: High-level overviews that answer "What is it?" and provide the background information necessary to understand Agent abstractions.
3. Tutorials: Broad, learning-oriented procedures that answer "Can you teach me to...?" These may use non-production-ready examples for educational purposes.
4. Tasks: Focused, objective-oriented guides that answer "How do I?" These provide production-ready solutions and best practices. They are designed to be followed step-by-step but may use branching paths instead of strict numbering.
5. Reference: The exhaustive source of truth describing Agent components and configuration options. This is the most detailed section and should not be paraphrased in Tasks or Tutorials; instead, those sections should link to the Reference pages.

Agent Flow is a system designed to describe a programmable pipeline for telemetry data. It moves away from a single hierarchical configuration file toward a model of reusable, composable components connected in a node graph or flowchart.

Key characteristics include:

• Component-based: The configuration is broken into discrete components (e.g., Service Discovery, Integrations, Filters, Remote Write).
• Programmable Pipeline: Components are connected to allow telemetry data to move through the system.
• Introspection: Designed to allow users to understand how data moves, how rules apply, and how filtering works.
• Visual/Machine Friendly: Aimed at being both machine-readable/writable (for GUIs and formatters) and visually representable as a node graph.

Conceptually, the data flow follows a pattern where a source (like Service Discovery) feeds into a filter, which feeds into an integration, which eventually reaches a destination (like Remote Write).

River is a declarative, HCL-inspired expression language designed specifically for Grafana Agent Flow. Unlike static configuration formats like YAML, River is an embedded language intended for continuous runtime evaluation.

Key Characteristics

• Declarative: Users define the desired state of the configuration, which maps naturally to configuration files.
• Expression-based: Settings for components are derived from expressions that can reference and mutate the outputs of other components.
• Go Integration: River natively supports passing around and invoking real Go values (such as interfaces and channels) and Go functions. This allows users to construct complex data pipelines using Go APIs without needing to understand the underlying implementation.
• Continuous Evaluation: Because it is an embedded language, the configuration can adapt dynamically to a changing environment at runtime.

Use Cases for Expressions

Instead of requiring dedicated components for every possible task, River expressions allow you to:

• Merge Metadata: Combine labels from distinct sources (e.g., merging Kubernetes namespace labels with pod labels).
• Chain Discoveries: Feed the output of one service discovery mechanism (like Kubernetes Service Discovery) into another (like HTTP Service Discovery).
• Conditional Logic: Implement custom logic, such as adjusting rate limits based on specific timeframes or business logic.

The Grafana Agent Operator is a Kubernetes operator designed for the static mode of Grafana Agent. It automates the deployment and configuration of Grafana Agents to collect telemetry data from Kubernetes resources.

Supported Telemetry Collection

The operator consumes several custom resources to discover and collect data:

• ServiceMonitor (Prometheus Operator): Collects metrics from Kubernetes Services.
• PodMonitor (Prometheus Operator): Collects metrics from Kubernetes Pods.
• Probe (Prometheus Operator): Collects metrics from Kubernetes Ingresses.
• PodLogs (Custom resource): Collects logs from Kubernetes Pods.

Key Limitations and Notes

• No Traces: The Grafana Agent Operator does not collect traces.
• Beta Status: The operator is currently in Beta and is subject to changes or removal.
• Grafana Cloud Users: If you are sending data to Grafana Cloud, it is recommended to use the Kubernetes Monitoring solution instead, as it provides a simplified setup with preconfigured dashboards and alerts.

By default, otelcol.exporter.loki does not convert OTLP attributes into Loki labels. To enable this conversion, you must inject specific "hint" attributes into the OTLP log using an otelcol.processor.attributes component:

• Resource Attributes: To convert OTLP resource attributes to Loki labels, use the loki.resource.labels hint attribute.
• Log Attributes: To convert OTLP log attributes to Loki labels, use the loki.attribute.labels hint attribute.

Label Translation: Labels are translated into a Prometheus-compatible format. For example, host.name becomes host_name and 2 cents becomes key_2_cents.

otelcol.processor.attributes "default" {
  action {
    key = "loki.attribute.labels"
    action = "insert"
    value = "event.domain, event.name"
  }
  
  action {
    key = "loki.resource.labels"
    action = "insert"
    value = "service.name, service.namespace"
  }
  
  output {
    logs = [otelcol.exporter.loki.default.input]
  }
}