DOCSEARCHbyAlgolia
Submit Docs

Zymtrace Documentation

website·Indexed 32 days ago

https://docs.zymtrace.com

A GPU profiling and performance analysis platform featuring flamegraphs, Efficiency IQ, and multi-GPU profiling. The documentation covers installation of the Zymtrace Backend and Profiler, configuration for NVIDIA and AMD GPUs, SLURM and MIG-aware profiling, and integration with inference servers like vLLM and SGLang. It includes guides on RESTful APIs, RBAC, OIDC authentication, and storage setup using ClickHouse, S3, and MinIO.

Tokens
50.2K
Snippets
220
Records
356
Agent score
49%

What's inside Zymtrace

  1. Overview of Zymtrace GPU Profiler Architecture

    Zymtrace provides a heterogeneous GPU profiling solution supporting NVIDIA GPUs, AMD GPUs, and AWS Neuron accelerators. It enables visibility into CUDA kernels, disassembles CUDA SASS mnemonics, and exposes GPU stall reasons. A key feature is the correlation of GPU kernel activity back to the CPU stack traces that launched them, allowing developers to identify when the GPU is idle because it is waiting on the CPU.

  2. Overview of Zymtrace Symbolization

    Symbolization is the process of transforming raw hexadecimal instruction addresses collected by the Zymtrace whole-system profiler into meaningful source code locations (triplets of <file name, function name, source line>). Because Zymtrace reads raw memory addresses without runtime instrumentation, symbolization is required to make profiles interpretable. It handles frames from CUDA kernels, system libraries, OS daemons, and native libraries.

  3. Overview of Zymtrace Model Context Protocol (MCP) Server

    Zymtrace implements the Model Context Protocol (MCP), an open standard that allows AI agents to securely connect to external data sources. This enables developers to query flamegraphs, investigate performance regressions, and receive optimization recommendations for CPU and GPU workloads using natural language directly within their IDE or terminal.

  4. Overview of Zymtrace

    Zymtrace is an autonomous optimization layer for heterogeneous, multi-silicon AI fleets. It provides cluster-wide introspection from application code down to GPU instructions to identify performance bottlenecks in kernels, code paths, and hardware topology without requiring code changes, recompilation, or system restarts.

    Key capabilities include:

    • Multi-Silicon Support: Supports NVIDIA (CUDA, NVLink), AMD (ROCm, Instinct), and AWS Neuron (Trainium, Inferentia). Support for Google TPUs and Cerebras is planned.
    • GPU Topology Health: Monitors NVLink health (CRC, replay, recovery), ECC/XID errors, and NUMA affinity.
    • Full-Stack Profiling: Continuous profiling of all processes (compiled and interpreted) across CPU/GPU boundaries using eBPF.
    • Inference & Training Optimization: Supports engines like vLLM, SGLang, and distributed training runs, attributing data to SLURM jobs, MIG instances, and Kubernetes namespaces.
    • Agentic Optimization: Exposes flamegraphs and GPU metrics via MCP (Model Context Protocol) for coding agents to identify bottlenecks and suggest fixes.

  5. Overview of Zymtrace AI-Assisted Profiling

    Zymtrace provides context-aware AI analysis for CPU and GPU profiling data. It allows developers to use natural language to identify bottlenecks, receive optimization recommendations (such as batching strategies or kernel fusion), investigate regressions across deployments, and interpret complex flamegraphs. The AI has access to profile metadata (hosts, namespaces, pods), flamegraph structures (call stacks, timing), GPU-specific patterns (CUDA kernels, memory copies), and historical context.

  6. Overview of SLURM-Aware GPU Profiling

    Zymtrace automatically tags every GPU kernel, CPU sample, and metric with SLURM job attributes, allowing distributed jobs spanning multiple nodes and GPUs to be viewed as a single entity. This attribution applies to GPU flamegraphs (CUDA/HIP kernels), GPU and host metrics (utilization, memory, power), and CPU profiles across various languages (Python, Go, Java, etc.).

  7. Overview of Zymtrace Profiler components

    The Zymtrace profiler consists of two primary components used for system and GPU analysis:

    1. zymtrace profiler: A host agent based on the OTel eBPF agent. It manages BPF unwinders and implements CPU profiling.
    2. zymtrace cuda profiler: A GPU profiler library that is loaded into CUDA workloads to trace performance issues like kernel stalls and memory bottlenecks, mapping them back to PyTorch code, GPU kernels, or scheduler threads.

  8. Verify system requirements for Zymtrace installation

    Before installing Zymtrace, ensure the host meets these hardware and kernel requirements:

    Architecture & Kernel:

    • CPU Architectures: x86_64 and ARM64
    • Minimum Kernel Version: 5.10 or higher (for both AMD64/x86_64 and ARM64)
    • Requirement: The kernel must be compiled with BPF Type Format (BTF) support.

    GPU Compatibility:

    • NVIDIA GPUs: CUDA 12.x or above
    • AMD GPUs: ROCm 7.x or above (x86_64 only)

    Privileges:

    • The profiler requires CAP_SYS_ADMIN capability. It must run as the root user or within a privileged container (e.g., in Kubernetes).
    # Check kernel version
uname -a

# Verify BTF support (check if this file exists)
ls /sys/kernel/btf/vmlinux

  9. Recommended Zymtrace performance analysis workflow

    Follow this three-step process to effectively analyze system performance:

    1. Initial Overview: Start with 'Efficiency IQ' to identify top resource-consuming functions and entities. Toggle between absolute and relative CPU modes depending on the goal.
    2. Refine Focus: Apply entity filters (e.g., specific pods or environments) using either click-and-apply or CEL query mode to isolate the problem area.
    3. Deep Dive:
      • For code optimization: Use Flamegraphs or Top Functions (Sandwich View).
      • For infrastructure decisions: Use Top Entities, then drill down into Flamegraphs or Top Functions.
      • For release validation: Use the Diff tool to compare states.

  10. Symbolization options in Zymtrace

    Zymtrace provides three ways to handle symbolization for native code applications:

    1. Automatic Upload: Zero-configuration, immediate symbolization.
    2. Global Symbolization: Provides extensive coverage for common system components and libraries.
    3. Manual Upload: Recommended for proprietary software, CI/CD integration, or when guaranteed symbol quality is required.

  11. Zymtrace Core Analysis Views Overview

    Zymtrace provides five primary views for performance analysis:

    • Efficiency IQ: A high-level dashboard showing overall compute utilization, top 5 expensive functions, and top 5 resource-consuming entities (hosts, deployments, containers).
    • Top Functions: Function-level analysis with CPU and GPU views, and a 'Sandwich View' to see callers and callees. Supports filtering by language, service, or infrastructure.
    • GPU & CPU Flamegraphs: Interactive call stack visualizations supporting mixed-stack views, panning, zooming, and grouping by executable, script, host, or thread.

  12. Zymtrace Backend Storage Architecture Overview

    Zymtrace relies on three primary storage backends to function:

    1. ClickHouse: The primary analytics database used for storing profiling data and metrics. Optimized for high-performance columnar analytical queries.
    2. Postgres: Handles identity management, user authentication data, and debugging symbol metadata.
    3. S3-Compatible Object Storage: Used for storing large binary files, such as debugging symbols and other artifacts.

    For each component, Zymtrace offers two deployment modes:

    • Create Mode (create): Deploys and manages the storage service directly within your cluster. Best for quick setup, development, and testing.
    • Existing Mode (use_existing): Connects to your own existing infrastructure (e.g., ClickHouse Cloud, AWS Aurora, GCP Cloud SQL, AWS S3, or MinIO). Best for production environments with established data governance.