repository·main·Indexed 19 days ago
https://github.com/elastic/apm-agent-goThe official package for instrumenting Go applications with Elastic APM to trace operation execution and send performance metrics and errors to an Elastic APM server. It provides instrumentation for Go kit, GORM (via apmgormv2), and database drivers (via apmsql). The agent collects built-in Go runtime, system, and process metrics and supports configuration via environment variables or Central Configuration. Note: This repository is in maintenance mode; migration to the OpenTelemetry Go API and SDK is recommended.
net/http and database/sql).In Go, the context.Context object is the primary mechanism for propagating request-scoped values, including APM transactions and spans, across function calls and goroutines.
To ensure spans are correctly parented to a transaction or a parent span, you must:
apm.ContextWithTransaction to add a transaction to a context. (Note: Middleware like apmhttp.Wrap does this automatically for HTTP requests).context.Context object into subsequent method calls.apm.StartSpan(ctx, ...) which returns a new context containing the newly created span. This new context should be used for any subsequent calls to ensure a proper hierarchy (e.g., Transaction -> Span A -> Span B).If a context does not contain a transaction or a span, any spans created with apm.StartSpan using that context will be dropped and not reported to the APM Server.
// 1. Start a span using the existing context
// apm.StartSpan returns the new span AND a new context containing that span
span, ctx := apm.StartSpan(ctx, "operation_name", "span_type")
defer span.End()
// 2. Pass the NEW context (ctx) to downstream functions to maintain the hierarchy
err := downstreamFunction(ctx)Trace context is used to correlate events across different services. It is based on the W3C Trace Context standard and contains:
This context is typically propagated between processes via HTTP headers, allowing the APM server to reconstruct a single distributed trace from multiple service calls.
The Elastic APM Go agent provides built-in instrumentation modules for various web frameworks and protocols. For each server instrumentation module, a transaction is reported for every handled request.
This transaction is stored in the request's context. You can retrieve this context using the specific API of your framework (e.g., http.Request.Context() for standard library or gin.Context.Request.Context() for Gin). Once you have the context, you can use it to report custom spans that are linked to the current transaction.
To correlate logs from your application with transactions captured by the Elastic APM Go Agent, your logs must include specific identifiers. This allows you to navigate between logs and traces in the Elastic observability stack.
Required Trace Identifiers:
transaction.idtrace.idspan.idRequired Service Metadata: To correlate logs to the correct service and environment, logs should also contain:
service.nameservice.versionservice.environmentCertain configuration options (like timeouts or buffer sizes) require specific unit formats. Units must be provided as a suffix directly after the number without whitespace.
Used for timeouts. Supported units:
ms (milliseconds)s (seconds)m (minutes)Example: 5ms
Used for maximum buffer sizes. The agent uses the power-of-two convention (e.g., 1KB = 1024B). Supported units:
B (bytes)KB (kilobytes)MB (megabytes)GB (gigabytes)Example: 10KB
The agent uses instrumentation modules to record events via middleware or wrappers.
http.Client or http.Transport using the module/apmhttp module.module/apmsql module, which provides instrumentation for well-known database drivers.To connect transactions with related spans and errors, and to propagate traces between services (distributed tracing), the agent relies on Go's built-in context package. Transactions and spans are stored within context objects. For incoming HTTP requests, trace data is recorded in the context object accessible via net/http.Context.
The APM Go Agent is part of a larger observability stack. To function correctly, it works in conjunction with:
Transaction.Sampled method. If it returns false, skip adding extra context or spans.When apmot is imported, transactions and spans created with the native Elastic APM API are made available as OpenTracing spans. This allows you to interleave both APIs in a single trace.
Important Note on Span Wrappers:
When using opentracing.SpanFromContext to retrieve a span created by the native API, the returned opentracing.Span is a wrapper intended only for context propagation. The following methods on these specific wrapper objects are no-ops:
Finish()Log*()Tracer()// Transaction created through native API.
transaction := apm.DefaultTracer().StartTransaction("GET /", "request")
ctx := apm.ContextWithTransaction(context.Background(), transaction)
// Span created through OpenTracing API will be a child of the transaction.
otSpan, ctx := opentracing.StartSpanFromContext(ctx, "ot-span")
// Span created through the native API will be a child of the span created
// above via the OpenTracing API.
apmSpan, ctx := apm.StartSpan(ctx, "apm-span", "apm-span")The Elastic APM Go Agent provides Log Correlation, which automatically injects correlation IDs into your application logs. These IDs enable seamless navigation between logs, traces, and services within the Elastic Observability stack.
To benefit from this, you can use one of two approaches: