DOCSEARCHbyAlgolia
Submit Docs

Pyroscope Go SDK

repository·main·Indexed 41 days ago

https://github.com/grafana/pyroscope-go

A Go integration for Pyroscope, an open-source continuous profiling platform. It enables the collection of CPU and memory profiling data via Push or Pull modes. The SDK includes godeltaprof for efficient delta profiling of heap, mutex, and block profiles, as well as a specialized HTTP handler to prevent CPU profiling conflicts with standard pprof endpoints.

Tokens
6.2K
Snippets
21
Records
27
Agent score
51%

What's inside Pyroscope Go

  1. Use the Pyroscope Go SDK k6 extension

    main
    The k6 package provides extension functions to enable integration between Pyroscope and k6. It specifically provides HTTP and gRPC middleware designed to dynamically label profiling contexts with k6 test metadata. This allows you to correlate profiling data with specific k6 test runs and metadata.

  2. What is godeltaprof and why use it

    main

    In Go, standard runtime/pprof profiles for memory, mutex, and block are cumulative, meaning they only grow over time and show all activity since the program started. This can result in "huge profiles" (megabytes in size) that are difficult to analyze.

    godeltaprof is an efficient delta profiler that computes the difference between two points in time. Instead of downloading two massive profiles and subtracting them locally, godeltaprof computes the delta on the raw records before serialization. This results in significantly smaller profile sizes (often reducing megabytes to tens of kilobytes) and lower CPU/memory overhead compared to traditional methods.

  3. Profile Go applications using Pull Mode

    main

    Pull Mode allows you to profile applications without adding the Pyroscope SDK code. To enable this, your application must expose standard Go profiling routes (/debug/pprof). This is typically achieved by importing the net/http/pprof package in your HTTP server setup.

    import _ "net/http/pprof"

  4. Compare godeltaprof with standard pprof and fastdelta

    main

    When choosing a profiling strategy, consider the following trade-offs:

    1. Standard runtime/pprof (Delta via seconds argument):

      • How it works: Dumps profile p0, sleeps, dumps profile p1, then subtracts p0 from p1.
      • Drawbacks: Requires two full profile dumps, produces high GC pressure, and can corrupt "in-use" values in heap profiles because it doesn't distinguish between allocation and in-use values during subtraction.

    2. DataDog's fastdelta:

      • How it works: Keeps a copy of the previous profile in memory and subtracts the current profile from it using a custom protobuf parser.
      • Drawbacks: Still requires parsing large (megabyte-scale) profiles before discarding most of the data.

    3. godeltaprof:

      • How it works: Computes the delta on raw runtime.MemprofileRecord and BlockProfileRecord before any serialization occurs. All zero values are rejected before the profile is even built.
      • Benefits: Much faster, produces significantly smaller serialized profiles, and avoids the overhead of parsing huge protobuf files.

  5. Use the Pyroscope CPU Profiler HTTP Handler to avoid profiling conflicts

    main

    The github.com/grafana/pyroscope-go/http/pprof package allows you to collect CPU profiles via HTTP without disrupting the background operation of the Pyroscope profiler.

    Standard Go pprof endpoints (like /debug/pprof/profile) return an error if profiling is already active: Could not enable CPU profiling: CPU profiling already in use. This handler resolves that by communicating with the Pyroscope profiler to coordinate profile collection.

    Important Implementation Note: To avoid panics, do not register this handler on the standard /debug/pprof/profile path or use the default http.DefaultServeMux if it might conflict with the standard net/http/pprof package. It is highly recommended to use a custom path such as /debug/pprof/cpu.

    When the handler is invoked, it performs the following lifecycle:

    1. Suspends the Pyroscope profiler.
    2. Gathers a CPU profile.
    3. Dispatches the profile to both the HTTP caller and the Pyroscope profiler.
    4. Resumes the Pyroscope profiler.
    package main

import (
    "net/http"

    "github.com/grafana/pyroscope-go/http/pprof"
)

func main() {
    // Use a custom path like /debug/pprof/cpu to avoid conflicts with standard pprof
    http.HandleFunc("/debug/pprof/cpu", pprof.Profile)
    http.ListenAndServe(":8080", nil)
}

  7. Profile a Go application using Push Mode

    main

    To use Push Mode, call pyroscope.Start() with a pyroscope.Config object. This sends profiling data directly to the Pyroscope server.

    Key configuration options include:

    • ApplicationName: A unique identifier for your application.
    • ServerAddress: The URL of your Pyroscope server.
    • Logger: The logger to use (set to nil to disable logging; pyroscope.StandardLogger is the default).
    • BasicAuthUser and BasicAuthPassword: Credentials for HTTP Basic authentication (required for Grafana Cloud).
    • TenantID: The Pyroscope tenant ID (required for multi-tenancy, not needed for Grafana Cloud).
    • ProfileTypes: A slice of pyroscope.ProfileType to specify which profilers to enable (e.g., pyroscope.ProfileCPU, pyroscope.ProfileAllocObjects, etc.). All profilers are enabled by default.
    package main

import "github.com/grafana/pyroscope-go"

func main() {
  pyroscope.Start(pyroscope.Config{
    ApplicationName: "simple.golang.app",

    // replace this with the address of pyroscope server
    ServerAddress:   "http://pyroscope-server:4040",

    // you can disable logging by setting this to nil
    Logger:          pyroscope.StandardLogger,

    // Optional HTTP Basic authentication (Grafana Cloud)
    BasicAuthUser:     "<User>",
    BasicAuthPassword: "<Password>",
    // Optional Pyroscope tenant ID (only needed if using multi-tenancy). Not needed for Grafana Cloud.
    // TenantID:          "<TenantID>",

    // by default all profilers are enabled,
    // but you can select the ones you want to use:
    ProfileTypes: []pyroscope.ProfileType{
      pyroscope.ProfileCPU,
      pyroscope.ProfileAllocObjects,
      pyroscope.ProfileAllocSpace,
      pyroscope.ProfileInuseObjects,
      pyroscope.ProfileInuseSpace,
    },
  })

  // your code goes here
}

  8. How baggage-to-label transformation works

    main

    The k6 package provides a specific convention for converting OpenTelemetry baggage into Pyroscope profiling labels:

    1. Prefix Filtering: Only baggage members whose keys start with k6. are considered. Other baggage keys are ignored.
    2. Value Validation: Baggage members with empty values are skipped.
    3. Key Transformation: To ensure compatibility with label formats, dots (.) in the baggage key are replaced with underscores (_). For example, a baggage key k6.user.id becomes the label k6_user_id.
    4. Label Application: The resulting key-value pairs are converted into a pyroscope.LabelSet and applied to the current goroutine via pprof.

  9. Use HeapProfiler to track heap allocation deltas

    main

    The HeapProfiler is a stateful profiler designed to track the delta of heap allocations in Go programs. Unlike standard pprof.WriteHeapProfile, which accumulates data over the entire lifetime of a program, HeapProfiler provides a snapshot of changes in heap usage since the last profile was written. It is safe for concurrent use.

    To use it, initialize a profiler using NewHeapProfiler() or NewHeapProfilerWithOptions() and then call Profile(w io.Writer) to write the profile data to the provided writer.

    // Initialize the profiler
hp := godeltaprof.NewHeapProfiler()

// ... perform application logic ...

// Write the heap profile delta to a writer (e.g., an os.File or http.ResponseWriter)
err := hp.Profile(someWriter)
if err != nil {
    // handle error
}

  10. How BlockProfiler works for delta profiling

    main

    A BlockProfiler is a stateful profiler used to track goroutine blocking events or mutex contention in Go programs.

    Unlike standard pprof.Lookup functions which accumulate data over the entire lifetime of a program, BlockProfiler tracks the delta (the changes) since the last time a profile was written. This allows you to capture snapshots of activity between two specific points in time rather than seeing a lifetime aggregate.

    There are two types of profilers you can create:

    1. Mutex Profiler: Uses runtime.MutexProfile to track mutex contention.
    2. Block Profiler: Uses runtime.BlockProfile to track goroutine blocking events.

    BlockProfiler is safe for concurrent use and serializes access to its internal state using a sync.Mutex.

  11. Add tags (labels) to profiling data

    main

    You can attach tags (labels) to specific code execution paths to enable filtering in the Pyroscope UI. This is done by wrapping a function call with labels. You can use either the pyroscope.TagWrapper or the standard pprof.Do approach.

    // these two ways of adding tags are equivalent:
pyroscope.TagWrapper(context.Background(), pyroscope.Labels("controller", "slow_controller"), func(c context.Context) {
  slowCode()
})

pprof.Do(context.Background(), pprof.Labels("controller", "slow_controller"), func(c context.Context) {
  slowCode()
})

  12. Configure the Pyroscope Profiler with Config struct

    main

    The Config struct defines how the profiler connects to the server and how it collects data.

    Key Configuration Fields:

    • ApplicationName: The name of your application (e.g., backend.purchases).
    • ServerAddress: The URL of the Pyroscope server (e.g., http://pyroscope.services.internal:4040). This can be overridden by setting the PYROSCOPE_ADHOC_SERVER_ADDRESS environment variable.
    • Tags: A map[string]string of global tags applied to all profiles.
    • UploadRate: The frequency at which profiles are uploaded.
    • ProfileTypes: A list of profile types to collect. If empty, DefaultProfileTypes is used.
    • DisableGCRuns: If true, disables automatic runtime.GC runs between heap profile captures.
    • TenantID: Used for multi-tenancy in Phlare.
    • BasicAuthUser / BasicAuthPassword: Credentials for HTTP Basic Auth.
    • HTTPHeaders: Custom headers to include in requests.
    • HTTPClient: A custom remote.HTTPClient implementation.
    • Logger: A custom logger implementation.

    Deprecated Fields:

    • AuthToken: Use BasicAuthUser and BasicAuthPassword instead.
    • DisableAutomaticResets: To be removed.
    • DisableCumulativeMerge: Ignored.
    • SampleRate: Fixed at 100; not configurable.