dd-trace-rb

repository·master·Indexed 19 days ago

https://github.com/datadog/dd-trace-rb

The datadog gem is a Ruby client library providing APM (Application Performance Monitoring) and security visibility for Ruby applications, enabling developers to trace requests and identify performance bottlenecks.

Tokens
90.1K
Snippets
256
Records
331
Agent score
64%

What's inside dd-trace-rb

  1. Overview of Datadog Trace Client for Ruby

    master
    The datadog gem is Datadog's client library for Ruby. It provides tools for visibility into the performance and security of Ruby applications, helping developers identify bottlenecks and other issues through Application Performance Monitoring (APM).
  2. How GVL profiling works

    master

    Profiling the Ruby Global VM Lock (GVL) is achieved using the Ruby GVL instrumentation API (available in Ruby 3.2+). The profiler monitors thread events to identify when threads are waiting for the GVL versus when they are actively running.

    Key Concepts

    • Waiting for GVL: Occurs when a thread is ready to run but is waiting to acquire the GVL. This is triggered by the RUBY_INTERNAL_THREAD_EVENT_READY event.
    • Running/Runnable: Occurs immediately after a thread acquires the GVL. This is triggered by the RUBY_INTERNAL_THREAD_EVENT_RESUMED event.

    Thread States and Visualization

    • Thread State: "Waiting for GVL" is represented as a special thread state. This state overrides other states (e.g., if a thread is in sleep but wakes up to request the GVL, it will be marked as "Waiting for GVL").
    • Visualization: The "Waiting for GVL" state affects the timeline visualization in Datadog but does not affect the regular flamegraph, as flamegraphs do not currently represent thread states.
    • Sampling Logic: To account for unaccounted cpu/wall-time, the profiler creates two samples for every sampled "Waiting for GVL" event: one to represent the waiting period itself and one to account for the time between the previous sample and the start of the waiting period.
  3. How Telemetry Handles Events Submitted Prior to Worker Start

    master

    Unlike other components in dd-trace-rb, the telemetry component allows events to be submitted before its background worker has actually started.

    This capability is specifically designed to capture and report errors that occur during the Datadog.configure process. If errors occur during configuration, the telemetry component buffers these events and transmits them once the telemetry worker successfully starts.

  4. Manage Dynamic Instrumentation performance and rate limits

    master

    Dynamic Instrumentation includes safeguards to minimize performance impact on your application.

    Default Rate Limits

    Probes have built-in rate limits that can be overridden in the probe definition:

    • Non-capturing probes: 5,000 invocations per second.
    • Capturing probes (with snapshots): 1 invocation per second.

    Circuit Breaker

    To protect application performance, an automatic circuit breaker is active:

    • If a probe's execution overhead exceeds 0.5 seconds of CPU time, it is automatically disabled.
    • Once disabled by the circuit breaker, the probe must be recreated in the Datadog UI to be used again.
    • This CPU threshold can be configured globally.
  5. Understand what data is captured by Dynamic Instrumentation

    master

    Dynamic Instrumentation captures two types of data: Probe snapshots (captured when a probe fires) and the Symbol Database (uploaded at startup).

    Probe Snapshots

    When a probe fires, the following data is sent to Datadog:

    • Variable values: Local variables, method arguments, and return values (subject to capture depth and size limits). Values are automatically redacted if they match built-in rules or your custom redaction configuration.
    • Object class names: The class of each captured value.
    • Exception details (method probes only): The exception class name and the message passed to the constructor. Note that if the constructor argument is not a string, the message will appear redacted.
    • Stack traces: The call stack at the moment the probe fires.

    Symbol Database

    This is uploaded once at startup to power auto-completion in the Datadog UI. It contains:

    • Class, module, and method names.
    • Method parameter names (names only, not values).
    • Source file paths and line ranges.
    • File content hashes for source code version matching.

    Note: The Symbol Database does not contain runtime values or application data.

  6. Limitations of OpenTelemetry Tracing with Datadog APM

    master

    When the Datadog APM integration is active alongside OpenTelemetry, certain OpenTelemetry features are unsupported or behave differently:

    • Context propagation: Unsupported. Datadog distributed header format is used instead.
    • Span processors: Unsupported.
    • Span Exporters: Unsupported.
    • OpenTelemetry.logger: Special behavior. It is set to the same object as Datadog.logger.
    • Trace/span ID generators: Special behavior. ID generation is performed by datadog.
  7. How profiling sampling works at run-time

    master

    Profiling operates via two independent background threads:

    1. The Scheduler: Wakes up every 1 minute to flush Exporter results via HttpTransport.
    2. The CpuAndWallTimeWorker: The 'active' part of the profiler. It manages timers and tracepoints to trigger sampling.

    When a sample is triggered, a synchronous pipeline is executed:

    1. Collectors::CpuAndWallTimeWorker identifies the event (Timing, GC, or Allocations).
    2. Collectors::ThreadContext gathers event details (cpu-time, thread id, span id, etc.).
    3. Collectors::Stack gathers the stack trace.
    4. StackRecorder records the sample into the native libdatadog data structure.

    Control then returns up the stack through each component until it returns to the Ruby VM.

  8. Manual tracing with the 1.0 Trace Model

    master

    In 1.0, manual tracing via Datadog::Tracing.trace provides two objects to the block: a Datadog::Tracing::SpanOperation (as span) and a Datadog::Tracing::TraceOperation (as trace).

    The span object is similar to the old 0.x span but with restricted access to certain fields like context. The trace object models the trace itself and provides access to trace-level state and functions.

    ### New 1.0 ###
    Datadog::Tracing.trace('my.job') do |span, trace|
      # Do work...
      # span => #<Datadog::Tracing::SpanOperation>
      # trace => #<Datadog::Tracing::TraceOperation>
    end
  9. Configure MongoDB trace settings per connection

    master

    You can apply different tracing configurations to different MongoDB connections using the describes option. This option accepts a connection string or a regular expression. When a connection matches a rule, the latest matching rule is applied.

    If a block is provided to instrument, it yields a settings object for that specific pattern.

    Datadog.configure do |c|
      # Match by exact connection string
      c.tracing.instrument :mongo, describes: '127.0.0.1:27017', service_name: 'mongo-primary'
    
      # Match by regular expression
      c.tracing.instrument :mongo, describes: /localhost.*/, service_name: 'mongo-secondary'
    end
  10. Understand the deprecation lifecycle in dd-trace-rb

    master

    Deprecated API objects in dd-trace-rb are scheduled for removal in the next major version release.

    To manage migrations:

    1. Identify warnings: Deprecated objects receive a deprecation tag in a minor version release. This allows you to surface warnings in your development environment before the breaking change occurs.
    2. Safe usage: You can continue to use deprecated objects safely until the next major version is released.
    3. Plan updates: When you see deprecation warnings, plan to migrate to the recommended replacement before upgrading to the next major version of the gem.
  11. Use TraceSegment in the processing pipeline (v1.0+)

    master

    In version 1.0 and later, when using a trace processor in the processing pipeline via Datadog::Tracing.before_flush, the block now provides a Datadog::Tracing::TraceSegment object as the trace argument, rather than an Array[Datadog::Span]. This TraceSegment object can be directly mutated.

    ### New 1.0 ###
    Datadog::Tracing.before_flush do |trace|
       # Processing logic...
       trace # => #<Datadog::Tracing::TraceSegment>
    end
  12. Upgrade Distributed Tracing: Propagation API and Constants

    master

    Propagation API

    Datadog::Tracing::Propagation::HTTP has moved to Datadog::Tracing::Contrib::HTTP.

    Environment Variables

    DD_PROPAGATION_STYLE_INJECT and DD_PROPAGATION_STYLE_EXTRACT are now prefixed with DD_TRACE_.

    B3 Strategy Constants

    B3 strategy values are now case-insensitive. Use the following for constants:

    • b3 for PROPAGATION_STYLE_B3_SINGLE_HEADER (single header)
    • b3multi for PROPAGATION_STYLE_B3_MULTI_HEADER (multiple headers)