Datadog .NET APM Tracer

repository·master·Indexed 20 days ago

https://github.com/datadog/dd-trace-dotnet

Provides automatic and manual instrumentation for .NET applications to enable distributed tracing and performance monitoring. Includes support for custom instrumentation via the Datadog.Trace package, advanced attributes through Datadog.Trace.Annotations, and CI Visibility integration via Datadog.Trace.BenchmarkDotNet. Offers deployment options through the Datadog.Trace.Bundle and Datadog.FeatureFlags.OpenFeature NuGet packages for environments where system-level installers are not feasible.

Tokens
78.7K
Snippets
175
Records
299
Agent score
69%

What's inside dd-trace-dotnet

  1. Overview of Profiler Memory Measurement

    master

    The profiler includes memory measurement capabilities to monitor and optimize the memory usage of its internal components. This visibility is used for:

    • Memory monitoring: Tracking consumption in production environments.
    • Leak detection: Identifying unusual or unbounded growth patterns.
    • Optimization: Locating memory hotspots.
    • Debugging: Providing detailed breakdowns for troubleshooting.

    Memory footprint metrics are automatically emitted via DogStatsD for production monitoring, and a detailed breakdown is logged to the console during profiler shutdown.

  2. Overview of System.Memory types

    master
    The System.Memory package provides types for efficient representation and pooling of managed, stack, and native memory segments, as well as sequences of such segments. It also includes primitives for parsing and formatting UTF-8 encoded text within these memory segments. This is useful for high-performance memory management and text processing without unnecessary allocations.
  3. Overview of Datadog APM .NET Client Libraries

    master

    The dd-trace-dotnet repository provides client-side components for Datadog's Application Telemetry Collection and Application Performance Monitoring (APM) for .NET applications. It consists of two primary components:

    1. Datadog .NET Tracer: A set of libraries used to trace .NET code. It provides automatic instrumentation for supported libraries and supports custom instrumentation for your own code. It powers features like Distributed Tracing, Application Security Management, Continuous Integration Visibility, and Dynamic Instrumentation.
    2. Datadog .NET Continuous Profiler: Libraries designed to automatically profile your application to monitor performance.
  4. Use the Instrumentation Verification Library for diagnostics

    master

    The Instrumentation Verification library is a diagnostic tool used to triage cases where the Datadog CLR Profiler causes runtime exceptions such as BadImageFormatException, InvalidProgramException, TypeLoadException, ExecutionEngineException, or AccessViolationException. It works by generating an assembly on disk that contains all metadata and bytecode changes performed at runtime, allowing you to verify the correctness of the instrumented assembly using tools like PEVerify, ILVerify, and ILSpy.

    Use Cases

    • CI/CD: Verifying that new integrations or probes produce valid IL.
    • Diagnostics: Post-mortem diagnosis of crashes suspected to be caused by faulty instrumentation.
    • Development: Enabling breakpoint debugging of instrumented methods via tools like [dnSpy].
  5. Understand configuration precedence in Azure Functions

    master

    Azure Functions uses a hierarchy to resolve configuration settings. If a setting is defined in multiple places, the one higher in the list takes precedence.

    Host Configuration Hierarchy

    1. Environment Variables
    2. Application Settings (exposed as environment variables)
    3. host.json
    4. Worker Config Files (worker.config.json)
    5. Platform Defaults

    Worker Configuration Hierarchy

    1. Command Line Arguments (with switch mappings)
    2. Environment Variables (all)
    3. AZURE_FUNCTIONS_ Prefixed Environment Variables
    4. Default Values (in code)
  6. Understand Datadog APM for .NET support levels

    master

    Datadog APM for .NET provides different levels of support depending on the runtime and environment. Understanding these levels helps you determine the stability and feature availability of your instrumentation:

    • General Availability (GA): Full implementation of all features. Full support for new features, bug & security fixes.
    • Maintenance: Full implementation of existing features. Does not receive new features. Support for bug & security fixes only.
    • Beta: Initial implementation. May not yet contain all features. Support for new features, bug & security fixes provided on a best-effort basis.
    • Legacy: Legacy implementation. May have limited function, but no maintenance provided.
    • End-of-life (EOL): No support.
    • Unsupported: No implementation.
  7. Using Duck Chaining to access non-public nested types

    master

    Duck chaining allows you to interact with properties or methods that return non-public types by automatically wrapping those return values in a new duck-type proxy. This enables deep access into the internals of an object hierarchy.

    To implement duck chaining, define proxy interfaces for both the parent and the nested type. When the parent proxy's property returns the nested type, the library will automatically wrap it in the nested proxy interface.

    Example

    If MyHandler has an internal property Configuration of type MyHandlerConfiguration:

    public class MyHandler 
    {
        public string Name { get; set; } 
        internal MyHandlerConfiguration Configuration { get; }
    }
    
    internal class MyHandlerConfiguration 
    {
        public int MaxConnections { get; set; }
    }

    Define your proxies like this:

    public interface IProxyMyHandler
    {
        string Name { get; set; }
        IProxyMyHandlerConfiguration Configuration { get; }
    }
    
    public interface IProxyMyHandlerConfiguration
    {
        int MaxConnections { get; set; }
    }

    Calling proxyMyHandler.Configuration will return an instance of IProxyMyHandlerConfiguration, allowing you to access MaxConnections even though the underlying type is internal.

  8. Understand Azure Functions log file behavior

    master

    When troubleshooting Datadog instrumentation in Azure Functions, it is critical to understand how log files are managed:

    • Append-only & Persistent: Log files are not cleared on deployment. New entries are appended to the end, while old entries from previous versions remain at the beginning. Always filter by timestamp instead of using head or tail.
    • Host vs. Worker Processes (Isolated Mode):
      • Host Process: Manages triggers and scaling. Log pattern: dotnet-tracer-managed-Microsoft.Azure.WebJobs.Script.WebHost-{pid}.log.
      • Worker Process: Executes your code. Log pattern: dotnet-tracer-managed-dotnet-{pid}.log. Multiple worker processes (different PIDs) may exist.
    • Worker Lifecycle: Worker processes restart on deployment but may reuse the same PID (especially in Linux containers). This means a single log file can contain entries spanning multiple deployments and process lifetimes.
  9. Understand Duck-chaining exception timing

    master

    Duck-chaining occurs when a duck-type proxy uses another duck-type proxy as a property.

    Critical Warning: Unlike direct duck-proxy creation (which throws DuckTypeException when the proxy is created), duck-chained proxies throw exceptions only when the property is accessed.

    If IProxyRoot.Configuration is a duck-chained property and the underlying configuration object is invalid, the exception will not trigger during someObject.DuckType<IProxyRoot>(), but rather when you call proxy.Configuration.

    public interface IProxyMyHandler
    {
        string Name { get; set; }
        IProxyMyHandlerConfiguration Configuration { get; }
    }
    
    public int GetMaxConnections(object someObject)
    {
        // This succeeds even if Configuration is invalid
        var proxy = someObject.DuckType<IProxyMyHandler>();
     
        // ⚠ Exception is thrown HERE if Configuration is invalid
        var config = proxy.Configuration;
        
        return config.MaxConnections;
    }
  10. Optimize memory measurement using incremental tracking

    master

    For components with large internal structures (e.g., ManagedThreadList, FrameStore, DebugInfoStore, HeapSnapshotManager), traversing all items in GetMemorySize() can be expensive. Use incremental tracking to keep modifications $O(1)$.

    Implementation Pattern

    1. Add a cached size field (use std::atomic<size_t> for thread safety): mutable std::atomic<size_t> _cachedItemsSize;

    2. Update size during modifications:

      • In AddItem(): Calculate the new item's size and use _cachedItemsSize.fetch_add(itemSize, std::memory_order_relaxed);
      • In RemoveItem(): Calculate the item's size and use _cachedItemsSize.fetch_sub(itemSize, std::memory_order_relaxed); before erasing.
      • In Clear(): Reset _cachedItemsSize.store(0, std::memory_order_relaxed);
    3. Calculate total in GetMemorySize(): Combine the constant container overhead (calculated on-demand) with the cached items size:

    size_t MyComponent::GetMemorySize() const {
        std::lock_guard<std::mutex> lock(_mutex);
    
        size_t totalSize = sizeof(MyComponent);
    
        // Calculate container overhead on-demand (cheap, no iteration)
        totalSize += _items.bucket_count() * (sizeof(Key) + sizeof(Value) + sizeof(void*));
    
        // Add cached items size (updated incrementally at add/remove time)
        totalSize += _cachedItemsSize.load(std::memory_order_relaxed);
    
        return totalSize;
    }

    When to use

    • Use when: Components have large collections (>100 items) or GetMemorySize() is called frequently (e.g., for metrics).
    • Avoid when: Collections are small (<50 items) where traversal is cheap, or if the structure is rarely modified and rarely queried.