tracing-opentelemetry-instrumentation-sdk

While both provide middleware for Axum, they serve different purposes:

Best Practice: Use them together. Use tower_http::trace for human-readable logs and OtelAxumLayer for distributed trace correlation.

When using tracing-opentelemetry to interoperate with OpenTelemetry, be aware of these behaviors:

• Span Creation Timing: An OpenTelemetry span is only created when the tracing span is closed. Do not attempt to interact with the OpenTelemetry Span or SpanBuilder directly from inside a tracing span.
• Context and Trace ID: The OTel parent Context (and trace_id) is created on a NEW span or inherited from a parent. Because the parent context can be overwritten after creation, a tracing log might report no trace_id or a trace_id that hasn't been set yet during the NEW event.
• Defining Span Metadata: To define the kind, name, etc., of an OpenTelemetry span via tracing, use specific record names such as otel.name, otel.kind, etc.
• Field Initialization: Since fields in a tracing span should be defined at creation time, use tracing::field::Empty for fields that will be updated later during the span's lifecycle.

This SDK provides helpers to build OpenTelemetry (OTel) instrumentation using the tracing crate, adhering to OpenTelemetry Trace Semantic Conventions.

Instrumentation Patterns

Caller Side (e.g., making an HTTP/gRPC request):

1. Start a span with attributes (using tracing::field::Empty for values to be filled later).
2. Inject propagation data into the call (e.g., via headers).
3. Execute the call.
4. Update span attributes with the response (e.g., status code).

Callee Side (e.g., receiving an HTTP/gRPC request):

1. Extract propagated info from headers to create an OpenTelemetry Context.
2. Start a span with attributes.
3. Attach the extracted context as the parent of the new span.
4. Process the request.
5. Update span attributes with the response (e.g., status code).

To use the OTLP exporter with the examples-axum-otlp example (or similar OpenTelemetry-instrumented applications), you must configure the following environment variables based on your preferred transport protocol (gRPC or HTTP/protobuf).

For gRPC

Use these settings to export traces via gRPC:

• OTEL_EXPORTER_OTLP_TRACES_ENDPOINT: Set to your collector endpoint (e.g., http://localhost:4317).
• OTEL_EXPORTER_OTLP_TRACES_PROTOCOL: Set to grpc.
• OTEL_TRACES_SAMPLER: Set to always_on to ensure all traces are captured.

For HTTP

Use these settings to export traces via HTTP/protobuf:

• OTEL_EXPORTER_OTLP_TRACES_ENDPOINT: Set to your collector endpoint path (e.g., http://127.0.0.1:4318/v1/traces).
• OTEL_EXPORTER_OTLP_TRACES_PROTOCOL: Set to http/protobuf.
• OTEL_TRACES_SAMPLER: Set to always_on to ensure all traces are captured.

# For GRPC:
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="http://localhost:4317"
export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL="grpc"
export OTEL_TRACES_SAMPLER="always_on"

# For HTTP:
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="http://127.0.0.1:4318/v1/traces"
export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL="http/protobuf"
export OTEL_TRACES_SAMPLER="always_on"

Use axum-tracing-opentelemetry to provide distributed tracing for Axum applications. The middleware handles:

• Reading OpenTelemetry headers from incoming requests.
• Starting a new trace if no trace context is found.
• Attaching the trace to the tracing span.
• Creating an OpenTelemetry Span when the tracing span closes.

To use it, apply OtelAxumLayer to your router to start traces on incoming requests, and optionally apply OtelInResponseLayer to inject the trace context back into response headers.

use axum_tracing_opentelemetry::middleware::{OtelAxumLayer, OtelInResponseLayer};

// ... setup tracing using init-tracing-opentelemetry ...

fn app() -> Router {
    Router::new()
        .route("/", get(index)) 
        // include trace context as header into the response
        .layer(OtelInResponseLayer::default())
        // start OpenTelemetry trace on incoming request
        .layer(OtelAxumLayer::default())
        .route("/health", get(health))
}

The fake-opentelemetry-collector is a basic OpenTelemetry collector designed for testing what is collected by OpenTelemetry. It allows you to start a mock collector server, initialize a TracerProvider that points to it, and then inspect the exported spans to verify your instrumentation logic. This is particularly useful for snapshot testing with tools like insta.

use std::time::Duration;
use fake_opentelemetry_collector::{setup_tracer_provider, FakeCollectorServer};
use opentelemetry::trace::{TracerProvider, SpanKind};

#[tokio::test(flavor = "multi_thread")]
async fn demo_fake_tracer_and_collector() {
    // 1. Start the fake collector server
    let mut fake_collector = FakeCollectorServer::start()
        .await
        .expect("fake collector setup and started");

    // 2. Initialize the TracerProvider linked to the fake collector
    let tracer_provider = setup_tracer_provider(&fake_collector).await;
    let tracer = tracer_provider.tracer("test");

    // 3. Create and end a span
    let mut span = tracer
        .span_builder("my-test-span")
        .with_kind(SpanKind::Server)
        .start(&tracer);
    span.add_event("my-test-event", vec![]);
    span.end();

    // 4. Flush and shutdown the provider to ensure spans are sent
    let _ = tracer_provider.force_flush();
    tracer_provider
        .shutdown()
        .expect("no error during shutdown");
    drop(tracer_provider);

    // 5. Retrieve and inspect the exported spans
    let otel_spans = fake_collector
        .exported_spans(1, Duration::from_secs(20))
        .await;
    
    // Example assertion using insta
    // insta::assert_yaml_snapshot!(otel_spans, ...);
}

To enable OpenTelemetry metrics, enable the metrics feature in your Cargo.toml. This initializes a global SdkMeterProvider.

Recommended: Direct Instrument Creation

The preferred way to produce metrics is to create instruments directly using the OpenTelemetry API. This keeps metric cardinality independent of log volume and avoids the overhead of emitting log lines.

Optional: Tracing-Macro Bridge

If the metrics feature is enabled, a MetricsLayer is added. This allows you to emit metrics by using specific field names in tracing macros (e.g., tracing::info!(monotonic_counter.name = 1)). Use this only when you are already emitting a trace/log for that event, as it couples metric emission to log volume.

use init_tracing_opentelemetry::opentelemetry::{KeyValue, global};

let http_requests = global::meter("my-service").u64_counter("http_requests").build();

http_requests.add(1, &[KeyValue::new("method", "GET"), KeyValue::new("path", "/")]);