DOCSEARCHbyAlgolia
Submit Docs

Open Agent Management Protocol (OpAMP)

repository·main·Indexed 41 days ago

https://github.com/open-telemetry/opamp-spec

A vendor-agnostic network protocol specification for the remote management of large fleets of data collection agents, such as the OpenTelemetry Collector. OpAMP enables remote configuration, status and telemetry reporting, heartbeating, package management, and secure auto-updates. It supports WebSocket and plain HTTP transport mechanisms and provides Protobuf definitions for communication between agents and management servers.

Tokens
21.7K
Snippets
42
Records
79
Agent score
100%

What's inside OpAMP

  1. Overview of the OpAMP Specification

    main

    The Open Agent Management Protocol (OpAMP) specification defines the standard for managing OpenTelemetry agents. This repository contains the core specification documentation and the Protobuf definitions used for communication.

    Key resources include:

    • Specification: Detailed protocol behavior in specification.md.
    • Protobuf Definitions: Located in the proto/ directory.
    • Instrumentation Guidelines: Standalone guidance for instrumentation runtimes is available in opentelemetry-guidelines.md.

  2. What is the Open Agent Management Protocol (OpAMP)?

    main

    OpAMP is a vendor-agnostic network protocol designed for the remote management of large fleets of data collection Agents. It enables a management Server to monitor and control a heterogeneous fleet of Agents (such as OpenTelemetry Collector, Fluentd, etc.) through a single interface.

    Key capabilities include:

    • Remote Configuration: Dynamically update Agent settings from a central Server.
    • Status Reporting: Agents report properties like type, version, and operating system, allowing Servers to tailor configurations to specific Agent types.
    • Telemetry Reporting: Agents can report their own process metrics (e.g., CPU, RAM) and operational metrics (e.g., data processing rates) to an OTLP-compatible backend.
    • Heartbeating: Continuous signaling to confirm Agent availability.
    • Package Management: Management of downloadable, Agent-specific software packages.
    • Secure Auto-updates: Capabilities for both upgrading and downgrading Agents securely.
    • Credential Management: Handling of connection credentials, including client-side TLS certificate rotation and revocation.

  3. Revoke Agent access via Connection Settings

    main

    A Server can revoke an Agent's access by:

    1. Marking the corresponding connection settings as revoked.
    2. Disconnecting the Client.

    Subsequent connection attempts using the revoked credentials will be rejected. For own_metrics, own_traces, or own_logs destinations, the Server MUST also communicate the revocation to those specific destinations so they can begin rejecting the revoked credentials.

  4. How heartbeat negotiation works in OpAMP

    main

    Heartbeats are used to keep connections active and inform the server of the Agent's liveness. The negotiation follows this flow:

    1. Connection: The Agent connects to the Server and optionally sets the ReportsHeartbeat capability.
    2. Negotiation: If ReportsHeartbeat is set, the Server may respond via OpAMPConnectionSettings.heartbeat_interval_seconds with a desired interval (30s is recommended) or 0 to disable heartbeats.
    3. Execution: If enabled, the Agent MUST send a heartbeat (an AgentToServer message with the instance_uid field set) every period.
    4. Maintenance: The Agent continues heartbeating on the configured interval while alive.

    Note: An AgentToServer message containing other data also counts as a heartbeat. The Agent should count the interval from the last AgentToServer message sent.

  5. Understand the OpAMP transport protocol choice (WebSocket)

    main

    The OpAMP specification utilizes WebSocket as its primary transport mechanism. WebSocket is a bidirectional, message-oriented protocol that establishes connections via plain HTTP and then leverages the existing TCP connection for message delivery.

    Key characteristics of the OpAMP transport choice include:

    • Bidirectional Communication: Allows both the Server and the Agent to send messages independently.
    • Standardized: Based on RFC 6455, ensuring wide support across browsers, servers, proxies, and load balancers.
    • Efficiency: Unlike HTTP polling, WebSocket maintains an active connection that is ideal for the OpAMP lifecycle, where connections may remain idle for hours or days without the overhead of constant polling requests.

  6. How Agent and Server interoperability works

    main

    OpAMP relies on a capability-based exchange to ensure that Agents and Servers can communicate even if they implement different versions or subsets of the specification.

    Capability Discovery

    During the initial message exchange, both parties declare their supported capabilities using bit-fields:

    • Agent Capabilities: Communicated via the AgentToServer message.
    • Server Capabilities: Communicated via the ServerToAgent message.

    Crucial Rule: Once a peer learns that a capability is NOT supported, it MUST stop attempting to use that capability with that specific peer.

    Handling Future Capabilities

    OpAMP supports two types of extensions to maintain interoperability between current and future versions:

    1. Ignorable Capability Extensions: New functionality added via new fields in Protobuf messages. Older recipients will simply ignore these unknown fields during deserialization.
    2. Non-Ignorable Capability Extensions: For functionality that cannot be ignored, OpAMP uses reserved bits in the AgentToServer and ServerToAgent capability fields.
      • Clients and Servers MUST set these reserved bits to 0 when sending messages.
      • A newer implementation seeing a 0 in these bits knows the sender does not support the new capability and can adjust its behavior accordingly.

  7. Use Plain HTTP Transport for OpAMP

    main

    OpAMP supports a synchronous, half-duplex message exchange over plain HTTP. The Client acts as an HTTP client making POST requests to the Server, and the Server acts as an HTTP server responding to those requests.

    Protocol Behavior

    • Message Exchange: The Client sends AgentToServer Protobuf messages in the request body. The Server responds with ServerToAgent Protobuf messages in the response body.
    • Polling: Because HTTP is half-duplex, the Server cannot push messages to the Agent. The Client MUST periodically poll the Server by sending an AgentToServer message where only the instance_uid field is set.
    • Polling Interval: The default interval is 30 seconds, but it SHOULD be configurable on the Client. If the Client has previously accepted OpAMPConnectionSettings, it SHOULD use the value from OpAMPConnectionSettings.heartbeat_interval_seconds as the polling interval.
    • Concurrency: The Client MUST wait for a response to a pending request before making a new one. However, a new request can be made immediately after a response is received without waiting for a polling period.
    POST / HTTP/1.1
Content-Type: application/x-protobuf
OpAMP-Instance-UID: f81d4fae-7dec-11d0-a765-00a0c91e6bf6

<binary_protobuf_agent_to_server_message>

  8. Understand the OpAMP Communication Model

    main

    The Open Agent Management Protocol (OpAMP) defines how agents and servers communicate to manage telemetry agents. It supports two primary transport mechanisms:

    1. WebSocket Transport: A bidirectional communication channel. It includes specific rules for message formats, size limits, and exchange patterns.
    2. Plain HTTP Transport: A standard request-response mechanism, subject to specific message size limits.

    Communication is structured through specific message types exchanged between the Agent and the Server, such as AgentToServer and ServerToAgent messages, as well as error responses.

  9. Establish and manage OpAMP connections

    main

    OpAMP connections are established via HTTP(S). If using WebSocket transport, the connection is upgraded from HTTP.

    Connection Lifecycle:

    1. Establishment: After the connection is established, the Client MUST send its first status report and expect a response.
    2. Retries: If connection fails, the Client SHOULD use an exponential backoff strategy with jitter. The Client SHOULD honor any throttling responses from the Server.
    3. Restoration: If a WebSocket connection breaks unexpectedly, the Client SHOULD attempt to reconnect immediately, falling back to exponential backoff if reconnection fails.
    4. Duplicates: A Client instance SHOULD connect no more than once. To reconnect, the Client MUST send an agent_disconnect message, close the existing connection, and then attempt a new connection.

    Note on instance_uid: If a Server detects duplicate instance_uids (e.g., due to VM cloning), it SHOULD generate a new instance_uid and send it via the new_instance_uid field in the AgentIdentification response.

  10. Close OpAMP connections

    main

    The procedure for closing a connection depends on the transport type and which party initiates the closure.

    WebSocket Transport

    • Client Initiated: The Client MUST first send an AgentToServer message with the agent_disconnect field set. It MUST then send a WebSocket Close control frame.
    • Server Initiated: The Server MUST send a WebSocket Close control frame.

    Plain HTTP Transport

    While the Client is logically disconnected once the HTTP response is complete, the Client SHOULD include the agent_disconnect field in its last AgentToServer message when shutting down or changing connection settings. This allows the Server to mark the Agent as disconnected immediately rather than waiting for polling timeouts.

  11. Use instance_uid in CSR for Agent Identity Verification

    main

    To prevent Agents from impersonating others, implementations may include the Agent's instance_uid within the CSR fields.

    Security Requirements:

    • The Server MUST verify that the instance_uid in the AgentToServer message matches the instance_uid contained within the CSR fields.
    • If an Agent's instance_uid changes (e.g., via a Server instruction), the Agent must repeat the CSR flow using the new instance_uid in the CSR.
    • Note for Servers: An incoming CSR may request a certificate with a different instance_uid than the one used in the current connection's client certificate. This is a valid state for performing an identity change and must not be rejected.

  12. Understand Agent Packages (Addons/Plugins)

    main

    An Agent is composed of one or more packages.

    • Top-level package: Implements the primary functionality of the Agent. There is usually only one, and it may have an empty name.
    • Sub-packages: Also known as addons or plugins. These provide additional functionality.

    Key Rules:

    • Uniqueness: An Agent cannot have more than one package with the same name installed.
    • Scoping: File names are only unique within the scope of a specific package.
    • Installation: Packages can be installed locally or offered remotely by the Server. The Server can only offer packages if the Agent's AgentToServer.capabilities includes the AcceptsPackages bit.