OpenTelemetry Go Automatic Instrumentation

The OpenTelemetry Go Instrumentation project aims to provide an automatic instrumentation experience similar to Java, Python, and JavaScript, but specifically for Go's compiled nature.

Key characteristics include:

• No code changes required: Any Go application can be instrumented without modifying its source code.
• Broad compatibility: Supports Go version 1.12 and above. It also works with binaries that have been stripped of debug symbols (e.g., using go build -ldflags "-s -w").
• Standardized configuration: Uses OTEL_* environment variables following the OpenTelemetry Environment Variable Specification.
• Standardized data: Instrumented libraries follow the OpenTelemetry specification and semantic conventions.

To avoid unstable instrumentation caused by hard-coded struct field offsets, this project uses a tool called offsetgen.

Instead of relying on fixed offsets that change when struct definitions change, offsetgen extracts offsets by analyzing binaries using DWARF debug information. To support stripped Go binaries (which lack DWARF info), the project maintains a cache of offsets in offset_results.json. This allows instrumentation authors to request field locations by name rather than hard-coded offsets, ensuring stability across different versions of the Go standard library and selected third-party packages.

The OpenTelemetry Go auto-instrumentation provides built-in support for several Go standard library packages. In the rolldice example, the following packages are automatically instrumented to produce spans:

• net/http: Captures HTTP request/response details.
• database/sql: Captures SQL database queries.

Check the compatibility documentation for a full list of supported packages.

The agent allocates memory for accessing eBPF maps using process.Allocate(). This process involves:

1. Size Calculation: Calculating map size based on the OS page size and CPU number.
2. Attachment: Attaching to the target process using ptrace via process.newTracedProgram().
3. Syscalls: Executing mmap to create the map, madvise to prepare the kernel for upcoming reads, and mlock to lock the address into RAM.
4. Tracking: The resulting address is stored in an AllocationDetails object within the TargetDetails.

To find the correct instrumentation points, the agent performs process analysis via analyzer.Analyze():

1. Target Details: The agent opens proc/{pid}/exe to read Go build information, the Go version, dependencies, and instrumentable functions.
2. Symbol Location: The Analyzer receives a list of relevant symbols from the Manager (via manager.GetRelevantFuncs()).
3. Function Mapping: The agent uses binary.FindFunctionsStripped() or binary.FindFunctionsUnstripped() to determine the exact memory locations of these functions within the process, returning a list of binary.Func objects.

Because Go is a compiled language that translates directly to machine code, traditional runtime code injection (common in Java or Python) is not possible.

Instead, this project uses eBPF (extended Berkeley Packet Filter). eBPF allows the instrumentation to attach user-defined code to the execution of a process within the Linux kernel. This mechanism enables the collection of telemetry without requiring the application to be recompiled or modified.

When the agent starts, it identifies the target process using the findPID() function. This function accepts either:

1. A PID (Process ID).
2. An executable path run by the process.

If an executable path is provided, the agent uses a ProcessPoller to iterate through the host's /proc directory to find a process that matches the target binary's name.

The instrumentation abstracts argument retrieval via a get_argument() function to handle changes in the Go ABI (Application Binary Interface):

• Go 1.17 and above: Arguments are passed using machine registers. The instrumentation detects the Go version and reads arguments from registers.
• Go versions below 1.17: Arguments are passed on the stack. The instrumentation reads arguments from the stack.

This detection is transparent to instrumentation authors.

OpenTelemetry spans require start and end timestamps. While eBPF typically uses uretprobes to trigger code at the end of a function, uretprobes and Go have known compatibility issues.

To solve this, the instrumentation analyzes the target binary to detect all return statements within the instrumented functions. It then places a uprobe at the end of each specific return statement to collect the end timestamp, bypassing the need for standard uretprobes.

The Go auto-instrumentation agent is a single binary that uses eBPF to attach to a target Go process. It orchestrates three internal components to achieve this:

• Process Analyzer: Locates the target process and identifies library functions available for auto-instrumentation.
• OpenTelemetry Controller: Uses the OpenTelemetry Go SDK to export the collected telemetry data.
• Instrumentation Manager: Coordinates the flow of events from eBPF programs to the OpenTelemetry Controller.

The agent relies on the Cilium eBPF libraries for Go for low-level eBPF operations like loading programs and reading events.

The loading phase is triggered by instrumentation.Load(), which executes the following:

1. Mounting: The Manager mounts the target binary and uses bpffs.Mount() to create a subdirectory under /sys/fs/bpf for the executable.
2. Probe Loading: The Manager calls probe.Load() on every registered probe. This applies inject options (such as loading offsets via inject.WithOffset()) to the target.
3. Kernel Loading: The agent builds a Cilium CollectionSpec and calls LoadAndAssign() to load the eBPF maps and programs into the kernel, then establishes links for the relevant Uprobes.

To create unified traces that combine manually created spans (via the OpenTelemetry Go SDK) with automatically instrumented spans (via eBPF probes), the instrumentation agent performs two key steps:

1. Span Modification: The agent attaches a uprobe to the function responsible for manual span creation. It overrides the new span's trace_id and parent_span_id using the current active span information retrieved from the eBPF map.
2. Active Span Map Update: Once the manual span is created, the agent updates the eBPF map to mark this new span as the current active span. This ensures that subsequent automatic or remote spans correctly link to this manual span, maintaining a continuous trace.

Note: This integration relies on the implementation of context propagation via eBPF maps.