DOCSEARCHbyAlgolia
Submit Docs

OpenTelemetry eBPF Instrumentation (OBI)

repository·main·Indexed 19 days ago

https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation

OBI provides zero-code, lightweight telemetry collection for user-space applications using eBPF. It includes TCP-based protocol parsers for Aerospike, Couchbase, Kafka, Memcached, MQTT, NATS, AMQP, SunRPC, and PostgreSQL, as well as runtime metrics for Go and JVM. Features include HTTP header enrichment (v0.7.0) and route pattern aggregation for HTTP metrics.

Tokens
91.3K
Snippets
184
Records
321
Agent score
18%

What's inside OpenTelemetry eBPF Instrumentation

  1. Overview of OpenTelemetry eBPF Instrumentation (OBI)

    main

    OpenTelemetry eBPF Instrumentation (OBI) provides a lightweight and efficient way to collect telemetry data for user-space applications using eBPF, based on the OpenTelemetry standard. It enables zero-code instrumentation, allowing you to collect spans, metrics, and attributes without modifying application code.

    Note on Stability: OBI is currently in development (v0). Users should expect breaking changes between minor releases. For production evaluation, it is recommended to pin to a specific semver release tag rather than using latest and to review release notes before upgrading.

  2. Supported OBI TCP-based protocol parsers

    main

    OpenTelemetry eBPF Instrumentation (OBI) provides protocol parsers that derive telemetry from raw TCP packets. The following protocols are currently supported:

    • Aerospike: Aerospike native client protocol.
    • Couchbase: Couchbase using the Memcached Binary Protocol.
    • Kafka: Kafka protocol.
    • Memcached: Memcached text protocol.
    • MQTT: MQTT protocol.
    • NATS: NATS protocol.
    • AMQP: AMQP 1.0 protocol.
    • SunRPC: ONC RPC (Sun RPC) protocol.
    • PostgreSQL: PostgreSQL protocol.

  4. What is the Kubernetes metadata cache service (k8s-cache)?

    main

    The k8s-cache is an optional, standalone Go service designed to centralize Kubernetes metadata collection for multiple OpenTelemetry eBPF Instrumentation (OBI) instances.

    Instead of every OBI pod opening its own expensive LIST/WATCH connections to the Kubernetes API server, they all subscribe to k8s-cache via a gRPC stream. The service maintains an in-memory view of Pod, Node, and Service resources and replays snapshots to new clients.

    Key Benefits:

    • Reduced API Server Load: Only the cache service maintains long-running connections to the Kube API.
    • Lower Memory Footprint: Cluster metadata is stored once in the cache rather than being replicated across every OBI node/pod.
    • Faster Cold Starts: OBI instances receive a pre-populated snapshot immediately upon subscription.
    • Efficient Reconnects: OBI can request updates starting from a specific timestamp to minimize replay volume.

  5. Configure the trace correlation communication channel

    main

    The communication channel between OBI and a profiler is an eBPF map pinned at a specific path.

    By default, the map is pinned at $PINPATH/otel/traces_ctx_v1, where $PINPATH defaults to the BPF filesystem (/sys/fs/bpf).

    Important: If you configure a custom location for OBI, the profiler must be configured to use the exact same location to ensure they can access the same shared map.

  7. SunRPC (ONC RPC) Detection and Limitations

    main

    Detection

    Kernel eBPF classifies SunRPC connections (k_protocol_type_sunrpc) when the first TCP record contains a valid ONC RPC CALL or REPLY header (record marking, RPC version 2, known program range, valid auth flavor). If OBI attaches mid-connection, userspace fallback parsing in matchSunRPC handles the traffic.

    Limitations

    • Protocol: Supports TCP only (no UDP SunRPC).
    • Capture Requirements: Kernel classification requires a complete single-fragment TCP record in the captured buffer.
    • Security: RPCSEC_GSS hides procedure arguments; only header fields (program, version, procedure) are visible.
    • Context: No distributed context propagation is supported on SunRPC.
    • Mapping: Procedure names may not be mapped (procedure number only) depending on the version/extension state.

  8. How SQL++ Namespace Resolution works

    main

    The OBI parser extracts bucket and collection information from SQL++ queries using the following priority and logic:

    1. Table path in statement: e.g., SELECT * FROM bucket.scope.collection``.
      • Bucket: bucket
      • Collection: scope.collection
    2. query_context field: If present in the request body, it provides the default namespace (e.g., default:bucket.scope``). The bucket is extracted from this context if not explicitly defined in the table path.
    3. Single identifier:
      • If query_context is set: The identifier is treated as a collection name.
      • If query_context is NOT set: The identifier is treated as a bucket name (legacy mode).

  9. Understand the OBI PostgreSQL protocol parser

    main

    The OBI PostgreSQL protocol parser identifies and extracts telemetry from PostgreSQL connections by inspecting the frontend/backend protocol (version 3).

    Message Structure

    Every message exchanged after the startup handshake follows this shape:

    • Type (1 byte): A single byte representing the message type.
    • Length (4 bytes): A big-endian (network byte order) integer.
    • Body: The message payload.

    Important Note on Length: The length field includes its own 4 bytes but does not include the leading type byte. For example, a message with a declared length of 13 will occupy a total of 14 bytes on the wire. The minimum valid length is 4 (representing an empty body, such as a Sync message).

    Recognized Frontend Message Types

    OBI uses specific frontend command types to recognize a connection:

    • 'Q': Query (simple query)
    • 'P': Parse (extended query)
    • 'B': Bind (extended query)
    • 'E': Execute (extended query)
    [ 1 byte: message type ] [ 4 bytes: length ] [ ...body... ]
       e.g. 'Q'             big-endian Int32      payload

  10. Understand StatsO11y and statistical metrics

    main

    StatsO11y

    StatsO11y is the component responsible for calculating statistical metrics (e.g., TCP RTT or failed-connection counts) across all applications running on a node. Unlike application-specific instrumentation, these metrics are calculated regardless of which PID triggered the event.

    Key Characteristics:

    • Scope: Node-wide, covering all applications.
    • Implementation: Probes reside in bpf/statsolly/.
    • Attributes: Metrics use AttrReportGroup structures (defined in pkg/export/attributes/attr_defs.go) for both Kubernetes (statsKubeAttributes) and non-Kubernetes (statsAttributes) environments. Users can enable or disable specific attributes via configuration.
    • Separation of Concerns: StatsO11y is distinct from AppO11y and NetO11y because statistical metrics are often more useful when correlated to all applications on a node, and certain hook points used for stats can make reliable PID calculation difficult.

  11. How the OBI NATS protocol parser works

    main

    The OBI NATS protocol parser provides instrumentation for plain TCP NATS traffic. It operates on a line-oriented text protocol where frames start with a control line terminated by \r\n.

    Because NATS mixes control frames and message frames on the same connection (unlike request/response protocols), OBI scans buffered traffic in both directions. If a single TCP event contains both a client publish and a server-delivered message, OBI emits the publish span as the main span and a second span for the delivered message.

    Key behaviors:

    • Span Creation: Spans are only created for message-carrying commands. Control traffic (like INFO, CONNECT, SUB, etc.) is used for validation but does not generate spans.
    • Header Support: OBI supports both standard (PUB/MSG) and header-aware (HPUB/HMSG) frames. It extracts the subject and payload length to create spans.
    • Proxy Protection: To prevent misclassifying HTTP proxy CONNECT requests as NATS, OBI validates that CONNECT and INFO frames are followed by valid JSON.

  12. How OBI versioning works (v0 vs v1+)

    main

    OBI follows Semantic Versioning 2.0.0. The versioning behavior changes significantly once the project moves from v0 to v1.

    During the v0 phase (Current Status)

    In v0, the project is in active development and stability guarantees are not yet provided.

    • Minor version bumps (v0.X.Y): Used for incompatible changes to public packages, flags, configuration, defaults, telemetry shape, or support expectations.
    • Patch version bumps (v0.X.Y): Used only for backward-compatible fixes (security, documentation, packaging).
    • Pre-release tags: May use tags like -rc1 for release candidates.

    After v1 (Future Stability)

    Once OBI reaches v1, it will provide stable compatibility guarantees for explicitly declared stable surfaces. At that point, standard SemVer rules apply:

    • Major version (vX.0.0): Required for any breaking change to a stable surface (API changes, CLI changes, config changes, telemetry changes, or dropping support for a previously supported environment).
    • Minor version (v1.X.0): Allowed for backward-compatible additions (new APIs, new optional flags/configs, new telemetry, or broadening the support matrix).
    • Patch version (v1.X.Y): Allowed only for backward-compatible fixes that do not require user migration (bug fixes, security fixes, documentation).