Orchestrion

repository·main·Indexed 20 days ago

https://github.com/datadog/orchestrion

A tool for automatic compile-time instrumentation of Go applications. Orchestrion leverages the Go toolchain's -toolexec feature to automatically insert tracing and observability logic into application code, dependencies, and the Go standard library without manual boilerplate. It is vendor-agnostic, supporting providers like Datadog and OpenTelemetry via configuration in a specialized orchestrion.tool.go file.

Tokens
10.1K
Snippets
30
Records
48
Agent score
62%

What's inside Orchestrion

  1. What is Orchestrion?

    main
    Orchestrion is a tool designed to automatically add Datadog instrumentation to Go applications during the build process. It leverages the Go toolchain's -toolexec feature to intercept and modify compilation units (including application code, dependencies, and the Go standard library) before they are compiled or linked.
  2. What is Orchestrion and how does it work?

    main

    Orchestrion is a tool for automatic compile-time instrumentation of Go code. It processes Go source code during the compilation phase and automatically inserts instrumentation logic.

    The instrumentation is driven by the specific imports present in a file named orchestrion.tool.go located at the project's root. By including specific package paths in this file, you signal to Orchestrion which integrations should be automatically applied to your codebase.

  3. How Orchestrion works

    main

    Orchestrion intercepts the standard Go toolchain build process using the -toolexec flag. It specifically targets two toolchain invocations:

    1. go tool compile (to instrument .go source files during compilation).
    2. go tool link (to manage link-time dependencies introduced by instrumentation).

    To maintain build efficiency and correctness, Orchestrion uses a job server (based on the NATS protocol) that persists for the duration of the build. This job server ensures:

    • A package is built exactly once, even if it is a shared dependency between the application and injected packages.
    • Expensive operations, like resolving injected package code objects via golang.org/x/tools/go/packages, are centralized and performed only once per package.
    • Version information for build cache invalidation is computed centrally.
  4. How code injection works in Orchestrion

    main

    Orchestrion uses a process similar to Aspect-oriented Programming (AoP) to inject code. It combines:

    • Join Points: Specific locations in the code where modifications are needed.
    • Advice: The actual modifications to be applied.

    To optimize performance, Orchestrion uses heuristics to avoid evaluating all 100+ available aspects (from dd-trace-go.v1) against every file. It filters aspects based on:

    • Dependency Closure: It only considers instrumentation if the relevant package (e.g., net/http) is in the package's dependency tree.
    • Source Content: It checks for specific strings (e.g., //dd:span) before attempting to match an aspect.

    The injector performs a depth-first traversal of the Abstract Syntax Trees (ASTs), evaluating applicable join points and applying advice where they match.

  5. How Orchestrion loads configuration and integrations

    main

    Orchestrion processes configuration by recursively traversing the imports defined in orchestrion.tool.go.

    • Recursive Loading: When an integration package is imported, Orchestrion loads all sub-packages in a tree-like structure. For example, importing github.com/DataDog/dd-trace-go/orchestrion/all will transitively enable packages like ddtrace/tracer, contrib/net/http, and contrib/database/sql.
    • De-duplication: Packages are automatically de-duplicated, so transitive imports do not cause conflicts.
    • Configuration Files: Each package encountered during this loading step can contain an orchestrion.yml file. These YAML files act as the backbone for auto-instrumentation, defining how the codebase is modified.
  6. How Orchestrion manages build cache invalidation

    main

    Orchestrion influences the Go toolchain's build cache by appending metadata to the output of intercepted -V=full invocations. This ensures that if Orchestrion's configuration or the injected dependencies change, the Go toolchain recognizes the build as different and invalidates the cache.

    The appended version string follows this format: compile version go1.23.6:orchestrion@v1.1.0-rc.1;<base64-encoded-hash>

    The <base64-encoded-hash> is composed of:

    • The specific Orchestrion configuration being used.
    • Details about all packages that may be injected by the configured integrations (resolved via packages.Load).

    Note: This approach results in more cache invalidations than strictly necessary because the Go toolchain currently lacks a more granular way to influence build identifiers.

    compile version go1.23.6:orchestrion@v1.1.0-rc.1;<base64-encoded-hash>
  7. Key features of Orchestrion

    main

    Orchestrion provides several capabilities for managing observability in Go applications:

    • Exhaustive Instrumentation: Uses -toolexec to instrument not just your application code, but also dependencies and the Go standard library.
    • Directives for Control: Allows developers to influence observability data using special code comments (directives).
    • YAML Configuration: Supports custom instrumentation logic via simple YAML documents, which is useful for applying specific configurations to custom frameworks.
    • Unobtrusive Workflow: Automates the injection of observability code so developers do not have to manually instrument their business logic.
  8. The Orchestrion compilation process

    main

    When the Go toolchain invokes go tool compile, Orchestrion performs the following steps for each package:

    1. Registration: Registers the build with the job server to check if the package is already built (idempotency) or needs a new build.
    2. Instrumentation: If a new build is required, Orchestrion parses .go files and applies configured integrations using github.com/dave/dst to decorate the AST.
    3. Dependency Management:
      • Modified source files are written to the Go toolchain's working directory with //line pragmas to preserve original line information.
      • If integrations inject new packages, Orchestrion updates the -importcfg file to provide archives for these new dependencies.
      • For main packages, a synthetic source file is created containing import statements for all recorded link-time dependencies to ensure func init() functions are correctly registered.
    4. Execution: Invokes the actual go tool compile with the modified source files and updated -importcfg.
    5. Artifact Update: Uses go tool pack to add a link.deps file to the produced .a archive, listing all implied link-time dependencies.
  9. How compile-time integrations work via aspects

    main

    In Orchestrion, compile-time integrations are modeled using a concept called aspects. An aspect is the combination of two components:

    • Join point: A standardized description of the specific location in your code where instrumentation code should be injected.
    • Advice: One or more descriptions of the actual modifications or code changes to be applied at that join point.
  10. How trace context propagation works with `//dd:span`

    main

    To ensure traces are not split across goroutine boundaries, you must propagate the trace context. //dd:span annotated functions handle context propagation based on their arguments:

    1. context.Context: If the function accepts a context.Context argument, that context is used for trace propagation.
    2. *http.Request: If the function accepts a *http.Request argument, the request's context is used.
    3. Fallback: If neither is present, the function relies on goroutine local storage, which means traces may split when starting new goroutines unless the context is explicitly passed.

    To weave context into a child goroutine, pass the context.Context through the function call.

    package demo
    
    //dd:span
    func caller(ctx context.Context) {
      wait := make(chan struct{}, 1)
      defer close(wait)
    
      // Weaving the span context into the child goroutine by passing ctx
      go callee(ctx, wait)
      <-wait
    }
    
    //dd:span
    func callee(ctx context.Context, done chan<- struct{}) {
      done <- struct{}{}
    }
  11. Control instrumentation with directives

    main

    You can influence the observability data produced by Orchestrion by adding special directives directly in your Go source code.

    Common directives include:

    • //orchestrion:ignore: Used to skip instrumentation for specific code blocks.
    • //dd:span custom-tag:value: Used to add custom tags to spans.
    // Example of using a directive
    //dd:span custom-tag:value
    func MyFunction() {
        // ...
    }
  12. The Orchestrion job server

    main

    Because Orchestrion wraps many short-lived processes via -toolexec, it uses a persistent job server (communicating via the NATS protocol) to share state across the entire build.

    Key responsibilities of the job server include:

    • Version Computation: Calculating the version information used for build cache invalidation.
    • Package Resolution: Resolving package archives for injected dependencies during both compile and link phases.
    • Task Caching: Storing compile task results to prevent re-instrumenting and re-compiling packages that are shared between the original build and injected dependencies.