OpenTelemetry Erlang/Elixir

The opentelemetry_exporter is used to send traces and metrics to an OpenTelemetry Collector. It currently supports the Tracer protocol via grpc or http_protobuf. By default, it exports protobuf-encoded Spans to http://localhost:4318/v1/traces using HTTP.

{opentelemetry_exporter,
  [{otlp_protocol, grpc}, 
   {otlp_compression, gzip}, 
   {otlp_endpoint, "https://api.honeycomb.io:443"}, 
   {otlp_headers, [{"x-honeycomb-dataset", "experiments"}]}]}

The structure of OpenTelemetry Semantic Conventions has changed. All attributes are now organized under a common attribute registry and classified by stability:

1. Stable: Standard attributes.
2. Experimental (Incubating): Attributes that are subject to change.

Attributes are organized by attribute group and stability. Previous code patterns are kept in a deprecated status to facilitate migration, but it is recommended to move to the new organized structure.

The OpenTelemetry implementation for Erlang and Elixir is split into two primary components following the OpenTelemetry specification:

1. API (opentelemetry_api): Defines the interfaces for tracing and instrumentation. Your application code should only depend on the API. If the SDK is not present, the API functions as a no-op implementation.
2. SDK (opentelemetry): The actual implementation of the API. This should be included in your production release along with an exporter to process and send telemetry data.

To capture distributed traces, you should use officially supported instrumentation libraries (found in opentelemetry-erlang-contrib) rather than manual instrumentation where possible.

The metrics system is composed of several layers:

1. Meter Provider: The entry point (implemented as otel_meter_server in the SDK). It manages shared configuration and the Resource of the telemetry. Including the SDK ensures a default Provider is available.
2. Meter: Used to create instruments (implemented as otel_meter_default in the SDK). Most users interact with Meters indirectly via macros.
3. Instrument: The object used to capture data. Can be synchronous (immediate recording) or observable (callback-based).
4. Measurement: An individual data point consisting of a value and associated attributes.
5. Metric Reader: (Part of the SDK) Triggers the collection of metrics, including executing callbacks for observable instruments.

Samplers control the number of traces collected and sent to the backend. The sampling decision is made when a span starts, meaning only the initial attributes passed to with_span or start_span are available to the Sampler.

Built-in Sampler Types:

• always_on
• always_off
• traceidratio
• parentbased_always_on
• parentbased_always_off
• parentbased_traceidratio

Configuration Options:

To implement a custom sampler, implement the otel_sampler behaviour.

All core OpenTelemetry Applications use the otel module prefix (e.g., otel_trace, otel_meter).

Because Erlang uses a flat namespace, this prefix allows modules to move between different packages (such as from opentelemetry_api_experimental to opentelemetry_api) without requiring users to change their code. If you are using the latest version of an experimental API, your code will continue to work seamlessly once that API graduates to a stable package.

When a Span is made active (e.g., via with_span), it is automatically added to the Erlang/Elixir logger metadata under the key otel_span_ctx.

You can configure your logger formatter to include trace_id and span_id in your logs by accessing these values from the metadata.

%% Example logger configuration to include trace/span IDs
{kernel,
  [{logger_level, debug}, 
   {logger, 
    [{handler, default, logger_std_h, 
      #{formatter => #{template => [..., {otel_trace_id, ["trace_id=", otel_trace_id, " "], []}, {otel_span_id, ["span_id=", otel_span_id, " "], []}, ...]}}}]}]}]}.