Grafana Dskit

When a span is eligible for profiling, the tracer sets span_id and span_name as pprof labels. These labels are stored in the goroutine's local storage and inherited by subsequent child goroutines.

Querying Profiles

• span_name is available as a regular label for query expressions. To find code not covered by traces, use: {service_name="my-service",span_name=""}.
• Trace spans are identified by the pyroscope.profile.id attribute. This allows you to find specific spans in the trace view and fetch their associated profiles.

Important: The presence of the pyroscope.profile.id attribute does not guarantee a profile is available; stack trace samples might not be collected if the CPU time used is below the sample interval (10ms).

Profiler Initialization

This module does not initialize the pprof profiler itself. You must initialize profiling using either the runtime/pprof package or the Pyroscope client.

The Service model is a Go implementation of the Google Guava service pattern. It provides well-defined, observable states for long-running tasks, allowing for asynchronous startup/shutdown and dependency management between services.

Service States

• New: The initial state after instantiation. The service is ready to be started.
• Starting: Transitioned to via StartAsync. The service is performing initialization.
• Running: The service has successfully initialized and is performing its primary work (e.g., responding to requests, background tasks).
• Stopping: Transitioned to via StopAsync. The service is performing cleanup.
• Terminated: A terminal state reached after successful cleanup in the Stopping state.
• Failed: A terminal state reached if an error occurs during Starting, Running, or Stopping.

Lifecycle Flow

• Successful path: New $\rightarrow$ Starting $\rightarrow$ Running $\rightarrow$ Stopping $\rightarrow$ Terminated.
• Failure path: Any error in Starting, Running, or Stopping transitions the service to Failed.
• Note: Once in Terminated or Failed, a service cannot be restarted.

Memberlist zone-aware routing is an optional feature designed to reduce cross-Availability Zone (AZ) data transfer costs. It works by categorizing nodes into two specific roles: member and bridge.

Node Roles

• member: A standard application instance. It only performs gossip and push/pull operations with other nodes (both members and bridges) located within the same zone. This keeps the majority of data traffic local to the zone.
• bridge: A specialized instance that facilitates inter-zone communication. A bridge can communicate with nodes in its own zone and with bridges in other zones. It acts as the gateway that allows different zones to propagate messages to one another.

Inter-zone Propagation Logic

To ensure messages eventually reach all zones without flooding the network, bridges follow specific selection rules:

• Broadcast messages: When selecting $N$ nodes to broadcast a message, a bridge will always select at least one node from the pool of bridges in other zones. The remaining $N-1$ nodes are selected randomly from the local zone and other-zone bridges.
• Push/pull syncs: When a bridge initiates a sync, it always contacts a random bridge in another zone. If no bridges exist in other zones, it falls back to selecting a random node in its own zone.

Note that node probes (health checks) may still cross AZ boundaries, but they represent a minimal fraction of total data transfer.

The local package provides an example for building a local cluster of multiple processes using the loopback interface. This allows you to simulate a distributed ring on a single machine by binding different processes to different loopback IP addresses.

Steps to run a local cluster:

1. Build the binary:

   go build local.go

2. Start the first peer: Bind it to a specific loopback address:

   ./local -bindaddr=127.0.0.1

3. Start additional peers: To join the existing ring, start a new process and use the -join-member flag to point to the address of an existing peer:

   ./local -bindaddr=127.0.0.2 -join-member=127.0.0.1

4. Scale the cluster: You can continue starting peers with unique loopback bindaddr values and joining them to the cluster.

Verifying the cluster

• Ring Status Page: Access the ring page of any peer to see all members in the ring. For example: http://127.0.0.1:8100/ring.
• Memberlist Status Page: Check the memberlist status at http://127.0.0.1:8100/kv.

Using as a client

You can interact with the ring information in client mode:

./local -mode=client

# Build
go build local.go

# Start first peer
./local -bindaddr=127.0.0.1

# Start second peer and join the first
./local -bindaddr=127.0.0.2 -join-member=127.0.0.1

# Run in client mode
./local -mode=client

To automatically profile root spans (the initial local spans in a process), wrap your existing OpenTracing tracer using spanprofiler.NewTracer.

This method is efficient because it selectively records profiles for root spans only. All stack trace samples accumulated during the execution of child spans contribute to the root span's profile. For example, an HTTP request results in a single profile regardless of how many spans are within that trace.

Note: This approach only captures spans created within the same goroutine (or its children) as the parent. For asynchronous execution where context is passed to detached goroutines, you must use explicit profiling via spanprofiler.StartSpanFromContext.

import (
    "github.com/opentracing/opentracing-go"
    "github.com/grafana/dskit/spanprofiler"
)

func main() {
    // Initialize your OpenTracing tracer
    tracer := opentracing.GlobalTracer()
    // Wrap it with the tracer-profiler 
    wrappedTracer := spanprofiler.NewTracer(tracer)
    // Use the wrapped tracer in your application
    opentracing.SetGlobalTracer(wrappedTracer)

    // Or, as an oneliner:
    // opentracing.SetGlobalTracer(spanprofiler.NewTracer(opentracing.GlobalTracer()))
}