otelconnect-go

repository·main·Indexed 41 days ago

https://github.com/connectrpc/otelconnect-go

OpenTelemetry tracing and metrics instrumentation for Connect RPC servers and clients in Go. It provides a connect.Interceptor via NewInterceptor() to implement the OpenTelemetry rpc metrics and rpc spans specifications for unary and streaming RPC patterns.

Tokens
4.4K
Snippets
10
Records
21
Agent score
100%

What's inside otelconnect-go

  1. Reduce metrics and tracing cardinality

    main

    By default, otelconnect follows OpenTelemetry RPC conventions, which include tagging all metrics and trace data with the server's IP address and the remote port number. This can lead to high cardinality in your observability backend.

    To manage this, you can use the following options:

    • otelconnect.WithoutServerPeerAttributes: Drops the server's IP address and remote port attributes.
    • otelconnect.WithFilter: Provides more granular control for customizable attribute filtering.
  2. Configure tracing for internal services

    main

    By default, otelconnect instrumented servers behave as if they are internet-facing: they do not trust client-provided tracing information and create new top-level trace spans for each request (linking them to the remote span via trace.Link).

    If your server is an internal service and you want it to trust the client's tracing context and create child spans instead of new top-level transactions, use the otelconnect.WithTrustRemote option.

  3. Add OpenTelemetry tracing and metrics to Connect servers and clients

    main

    Use connectrpc.com/otelconnect to add OpenTelemetry support to your Connect-Go services. The primary way to achieve this is by creating an interceptor using otelconnect.NewInterceptor(). This interceptor can be applied to both Connect handlers (servers) and Connect clients.

    By default, the interceptor uses the global OpenTelemetry TracerProvider and MeterProvider. If you need to use specific providers instead of globals, use the following options:

    • otelconnect.WithTracerProvider
    • otelconnect.WithMeterProvider
    package main
    
    import (
    	"context"
    	"fmt"
    	"log"
    	"net/http"
    
    	"connectrpc.com/connect"
    	"connectrpc.com/otelconnect"
    	// Generated from your protobuf schema
    	pingv1 "connectrpc.com/otelconnect/internal/gen/observability/ping/v1"
    	"connectrpc.com/otelconnect/internal/gen/observability/ping/v1/pingv1connect"
    )
    
    func main() {
    	mux := http.NewServeMux()
    	// Create the interceptor
    	otelInterceptor, err := otelconnect.NewInterceptor()
    	if err != nil {
    		log.Fatal(err)
    	}
    
    	// Apply to a server handler
    	mux.Handle(pingv1connect.NewPingServiceHandler(
    		&pingv1connect.UnimplementedPingServiceHandler{},
    		connect.WithInterceptors(otelInterceptor),
    	))
    
    	http.ListenAndServe("localhost:8080", mux)
    }
    
    func makeRequest() {
    	// Create the interceptor
    	otelInterceptor, err := otelconnect.NewInterceptor()
    	if err != nil {
    		log.Fatal(err)
    	}
    
    	// Apply to a client
    	client := pingv1connect.NewPingServiceClient(
    		http.DefaultClient,
    		"http://localhost:8080",
    		connect.WithInterceptors(otelInterceptor),
    	)
    	resp, err := client.Ping(
    		context.Background(),
    		connect.NewRequest(&pingv1.PingRequest{}),
    	)
    	if err != nil {
    		log.Fatal(err)
    	}
    	fmt.Println(resp)
    }
  4. Implement payload interception for streaming connections

    main

    The otelconnect package provides internal interceptor types to observe request and response bodies in streaming connections.

    Streaming Client Interceptor

    When implementing a StreamingClientInterceptor, you can wrap a connect.StreamingClientConn. The interceptor uses the following function signatures to hook into the stream lifecycle:

    • receive func(any, connect.StreamingClientConn) error: Called when a message is received from the server.
    • send func(any, connect.StreamingClientConn) error: Called when a message is sent to the server.
    • onClose func(): Called when the response stream is closed via CloseResponse().

    Streaming Handler Interceptor

    When implementing a StreamingHandlerInterceptor, you can wrap a connect.StreamingHandlerConn. The interceptor uses the following function signatures:

    • receive func(any, connect.StreamingHandlerConn) error: Called when a message is received from the client.
    • send func(any, connect.StreamingHandlerConn) error: Called when a message is sent to the client.
  5. Create an OpenTelemetry interceptor with NewInterceptor

    main

    Use NewInterceptor to create a connect.Interceptor that adds OpenTelemetry metrics and tracing to your Connect handlers (servers) and clients.

    By default, it uses the global OpenTelemetry tracer and meter providers. You can customize its behavior using Option functions, such as:

    • WithTracerProvider: Provide a custom OpenTelemetry tracer provider.
    • WithMeterProvider: Provide a custom OpenTelemetry meter provider.
    • WithoutMetrics: Disable metrics collection.
    • WithoutTracing: Disable tracing.

    If any provided options are invalid, NewInterceptor will return an error.

    interceptor, err := otelconnect.NewInterceptor(
        otelconnect.WithTracerProvider(myTracerProvider),
        otelconnect.WithoutMetrics(),
    )
    if err != nil {
        // handle error
    }
  6. Filter instrumented requests and attributes

    main

    You can control which requests are instrumented or which attributes are included in your telemetry to manage performance and data volume.

    • WithFilter(filter func(context.Context, connect.Spec) bool): Only emits traces and metrics when the provided function returns true. The function must be safe for concurrent use.
    • WithAttributeFilter(filter AttributeFilter): Sets a global filter for all metrics and trace attributes.
    • WithoutServerPeerAttributes(): A convenience option that removes net.peer.port and net.peer.name from server trace/span attributes. This is useful for reducing high-cardinality data in server environments.
  7. Implement streaming handler tracing and metrics with WrapStreamingHandler

    main

    The WrapStreamingHandler method wraps a connect.StreamingHandlerFunc to provide OpenTelemetry instrumentation for streaming server handlers.

    Key behaviors:

    • Span Lifecycle: The span covers the entire duration of the streaming handler.
    • Metrics: It tracks the number of messages received (requests) and sent (responses), along with the total request and response sizes.
    • Propagation: It extracts trace context from the incoming request headers to continue existing traces.
    // Example of wrapping a streaming handler
    handler := func(ctx context.Context, conn connect.StreamingHandlerConn) error {
        // ... implementation
    }
    
    interceptor, _ := otelconnect.NewInterceptor()
    wrappedHandler := interceptor.WrapStreamingHandler(handler)
  8. Implement unary tracing and metrics with WrapUnary

    main

    The WrapUnary method allows you to wrap a connect.UnaryFunc to automatically record OpenTelemetry spans and metrics for unary RPC calls.

    It captures:

    • RPC duration
    • Request and response sizes (if using Protobuf)
    • Request and response counts
    • Status codes and error messages
    • Metadata/Headers (based on configured header keys)

    It automatically handles trace propagation for both clients and servers.

    // Example of wrapping a unary handler
    handler := func(ctx context.Context, req connect.AnyRequest) (connect.AnyResponse, error) {
        return connect.NewAnyResponse(resp), nil
    }
    
    interceptor, _ := otelconnect.NewInterceptor()
    wrappedHandler := interceptor.WrapUnary(handler)
  9. Configure RPC System semantic conventions

    main

    By default, instrumentation uses semantic conventions based on the protocol used (e.g., gRPC or ConnectRPC). You can force a specific system to ensure uniform telemetry across different protocols.

    • WithRPCSystem(system RPCSystem): Forces the use of the specified RPC system's semantic conventions.
    • ConnectRPCSystem: Uses ConnectRPC semantic conventions.
    • GRPCSystem: Uses gRPC semantic conventions.

    This is particularly useful when a single server handles multiple protocols but you want to aggregate metrics and spans uniformly.

  10. Inject a Labeler into a context with ContextWithLabeler

    main

    Use ContextWithLabeler to attach a *Labeler instance to a context.Context. This allows instrumented ConnectRPC handlers to access the labeler and add attributes that will be injected into emitted metrics. Note that only one labeler can be active in a context; calling ContextWithLabeler multiple times will override the previous labeler.

    ctx := otelconnect.ContextWithLabeler(parentCtx, myLabeler)