DOCSEARCHbyAlgolia
Submit Docs

OTel-Arrow (OTAP) Documentation

repository·main·Indexed 18 days ago

https://github.com/open-telemetry/otel-arrow

OTel-Arrow (OTAP) is a high-performance, column-oriented telemetry pipeline for OpenTelemetry data based on Apache Arrow, designed to reduce network, memory, and CPU usage compared to the standard row-oriented OTLP protocol. The documentation includes guides for running the OpenTelemetry Protocol with Apache Arrow bridge, metadata bridge, recorder, and shutdown test examples, as well as configuration details for the concurrent batch processor.

Tokens
387.3K
Snippets
772
Records
1.4K
Agent score
63%

What's inside OTel-Arrow

  1. Overview of the otap-df-admin interface

    main

    The otap-df-admin crate provides a management interface for the OTAP Dataflow Engine. It includes:

    • HTTP Endpoints: Provides admin, health, status, and telemetry data.
    • Live Pipeline Mutation: Endpoints to create, replace, resize, and track rollout or shutdown operations.
    • Embedded UI: A single-page application (SPA) served from the same process and origin as the API, allowing for visual monitoring and management.

    For detailed architecture, refer to docs/admin/architecture.md. For instructions on managing pipelines via the UI or API, see the operator guide for live reconfiguration.

  2. Overview of Plugin Development in the Test Orchestration Framework

    main
    The Test Orchestration Framework uses a modular, pluggable architecture to extend the test lifecycle. Plugins allow you to customize various phases of execution, including configuration, deployment, execution, monitoring, hooks, and reporting. Each plugin is type-safe, adheres to a specific interface, and is discovered at runtime via YAML configuration using a unique type name.

  3. What is Quiver?

    main

    Quiver is an experimental, standalone, and embeddable Arrow-based segment store provided as a Rust crate. It is designed for OTAP Dataflow to provide durable buffering and crash recovery for telemetry pipelines.

    Key features include:

    • Write-Ahead Log (WAL): Provides crash recovery with configurable flush policies.
    • Segment Storage: Uses immutable Arrow IPC files, supporting optional memory-mapped reads.
    • Multi-Subscriber: Supports multiple independent consumers with at-least-once delivery guarantees.
    • Progress Tracking: Uses persistent files to track subscriber state.
    • Automatic Cleanup: Automatically deletes segments once all subscribers have completed processing them.

  4. Overview of the Recordset Query Engine

    main
    The Recordset Query Engine is an implementation designed to operate on hierarchical data records. It provides a mechanism for generating summaries based on grouping criteria and time windows. The engine is agnostic to the underlying data model (such as OpenTelemetry) and simply returns processed data, allowing the caller to determine how to handle or store the output.

  5. Overview of OTel-Arrow Rust libraries

    main
    The OTel-Arrow Rust sub-projects provide tools for building OpenTelemetry pipelines using an Apache Arrow-first approach. The core of the Rust ecosystem is the otap-dataflow engine, which allows for high-performance telemetry processing by representing OpenTelemetry data as Arrow record batches.

  6. Overview of otap-df-pdata-views

    main

    The otap-df-pdata-views crate provides zero-dependency, backend-agnostic, read-only view traits for traversing hierarchical OTLP/OTAP telemetry data structures, including logs, traces, and resources.

    It is designed for two primary use cases:

    1. Internal consumption within the otap-dataflow workspace.
    2. Lightweight external integration for crates that need to interact with telemetry data without pulling in the full otap-df-pdata stack.

  7. Overview of the Azure Monitor Exporter

    main

    The Azure Monitor Exporter sends OpenTelemetry logs to Azure using the Azure Logs Ingestion API. It transforms OTLP log data into the format expected by Azure Log Analytics and provides configurable schema mapping for custom log tables.

    Key Characteristics:

    • Type: urn:microsoft:exporter:azure_monitor
    • Stability: Alpha; currently supports logs only.
    • Authentication: It does not manage credentials itself. Instead, it relies on the azure_identity_auth extension, consuming OAuth bearer tokens via the bearer_token_provider capability.
    • Schema Mapping: Allows mapping OTLP resource attributes, scope attributes, and log record fields to specific Azure columns. Unmapped fields are dropped by default.

  8. Overview of the OTAP Dataflow Engine

    main

    The OTAP Dataflow Engine is a collection of Rust crates designed to provide OpenTelemetry pipeline support as an embedded software component. It provides a framework for collecting OpenTelemetry data using the OpenTelemetry Protocol with Apache Arrow (OTAP).

    Key characteristics include:

    • OTel-Arrow First: Uses Apache Arrow record batches as the underlying data type for "OTAP records". Data is represented in a column-oriented "star schema" to handle hierarchical OpenTelemetry data.
    • Zero-copy OTLP Conversion: Uses a custom Google Protocol Buffers implementation to convert OTAP records directly to and from OTLP bytes without intermediate data objects.
    • Thread-per-core Architecture: Prioritizes single-threaded, nothing-shared pipelines using a local async runtime to avoid synchronization overhead. While multi-threaded components are possible via shared adapters, local components are preferred.

  9. Overview of OTel-Arrow Go libraries

    main

    The OTel-Arrow Go libraries provide a reference implementation for converting between OpenTelemetry (OTLP) data representations and the OTAP (OpenTelemetry Protocol with Apache Arrow) stream format.

    This implementation is designed to work with the OpenTelemetry Collector's in-memory pdata representation, specifically targeting the following top-level types:

    • ptrace.Traces (Traces)
    • pmetric.Metrics (Metrics)
    • plog.Logs (Logs)

    Primary Data Conversions:

    1. OTLP records to OTAP stream: Converts the Collector's in-memory data into an Arrow IPC stream.
    2. OTAP stream to OTLP records: Converts Arrow IPC stream records back into the standard Collector representation.

    Note: While an intermediate "OTAP records" format exists in this Go implementation, it is not the primary focus of the design. For detailed information on handling the OTAP records format in memory, refer to the Otel-Arrow-Rust implementation.

  10. Overview of report_formatters plugins

    main

    The report_formatters plugin suite provides different ways to format performance reports within the pipeline. You can choose between a no-op formatter, a JSON formatter, or a Jinja2-based template formatter depending on whether you need raw data, structured JSON, or custom-rendered text.

    | Type Name | Module | Class | Config Class | Description Summary |
| --------- | ------ | ----- | ------------ | -------------------- |
| `noop` | `lib.impl.strategies.common.report` | `NoopFormatter` | `NoopFormatterConfig` | A report formatter that performs no formatting and returns an empty string |
| `json` | `lib.impl.strategies.common.report` | `JsonFormatter` | `JsonFormatterConfig` | Formats a report as a JSON string using the specified configuration |
| `template` | `lib.impl.strategies.common.report` | `TemplateFormatter` | `TemplateFormatterConfig` | Formats a report using a Jinja2 template specified either by file path or inline string |

  11. Summary of available hook strategies

    main

    The hook_strategies plugin provides a variety of pre-defined hooks for managing component lifecycles, specifically focusing on Docker orchestration, process management, and execution control.

    Docker Strategies

    • build_docker_images: Builds multiple Docker images locally.
    • build_docker_image: Builds a single Docker image.
    • create_docker_network: Creates a Docker network if it doesn't exist.
    • delete_docker_network: Deletes a Docker network associated with a component.
    • tidy_existing_container: Removes an existing container with the same name as the component.
    • wait_for_status: Waits for a container to reach a specific status.

    Execution & Control Strategies

    • raise_exception: Raises an exception.
    • record_event: Records an event to the current span.
    • run_command: Runs a shell command.
    • send_http_request: Sends an HTTP request.
    • ready_check_http: Performs an HTTP(S) readiness check.
    • render_template: Renders a Jinja2 template.
    • ensure_process: Ensures a process is running and hasn't crashed.

  12. Use the Batch Processor (Deprecated)

    main

    Note: Component Status: Removed

    This component is no longer needed as the exporterhelper batcher is now mature. Use the exporterhelper instead.

    Purpose

    The batch processor accepts spans, metrics, or logs and places them into batches. Batching improves data compression and reduces the number of outgoing connections required for transmission. It supports both size-based and time-based batching.

    Best Practices

    • Placement: Define the batch processor in the pipeline after the memory_limiter and any sampling processors. This ensures batching occurs after any data drops (like sampling) to avoid unnecessary buffering.
    • Configuration: It is highly recommended to configure a batch processor on every collector.