DOCSEARCHbyAlgolia
Submit Docs

NVIDIA TensorRT-LLM

repository·main·Indexed 12 days ago

https://github.com/nvidia/tensorrt-llm

A high-performance inference optimizer for LLMs and Visual Generation models. It provides specialized kernels, such as FMHA_v2, and a customizable Pythonic framework for efficient model execution on NVIDIA hardware. Includes agent-flow, a torch-like framework for composing Claude and Codex agent layers, and specialized workflows for model bring-up.

Tokens
369.2K
Snippets
666
Records
1.3K
Agent score
95%

What's inside TensorRT-LLM

  1. Overview of custom kernel integration paths in TensorRT-LLM

    main

    TensorRT-LLM allows you to integrate custom GPU kernels into the PyTorch backend. A custom kernel is wrapped as a PyTorch custom op (accessible via torch.ops.trtllm.<name>). There are three primary integration paths depending on your source language and build requirements:

    FlavorSource languageBuild TimingLocation
    CUDA C++.cu / .hAt wheel build time (via CMake/nvcc)cpp/tensorrt_llm/kernels/
    CuTe DSLPython (cutlass.cute)JIT, on first call (cached in-process)tensorrt_llm/_torch/cute_dsl_kernels/
    cuTilePython (cuda.tile)JIT, on first call (cached in-process)
    tensorrt_llm/_torch/cuda_tile_kernels/

    Note: The trtllm-gen path (using pre-built CUBINs) is not covered in this guide.

  2. Overview of Sparse Attention in TensorRT LLM

    main

    TensorRT LLM provides a unified, extensible sparse attention framework designed to handle the computational and memory bottlenecks of long-context LLM workloads. The framework addresses two main phases:

    1. Context (Prefill) Phase: Reduces computational load to improve Time-to-First-Token (TTFT).
    2. Generation (Decode) Phase: Reduces KV cache memory footprint and bandwidth usage to improve Token-to-Token Latency (TPOT) and allow larger batch sizes.

    The framework is designed to be general enough to support different sparsity granularities (page-level vs. token-level), prediction strategies (heuristic vs. learned), and integration points (framework-driven vs. kernel-internal). It provides standardized abstractions such as prediction hooks, sparse indices, and metadata interfaces.

    Supported Algorithms:

    AlgorithmAttention TypeKey IdeaContext SupportGeneration Support
    RocketKVMQA/MHA/GQAKV cache eviction + dynamic Top-KKV cache compressionSparse computation
    DSAMLANeural indexer + sparse MLA kernelSparse computationSparse computation
    BLASSTMQA/MHA/GQA/MLADynamic block skipping in kernelN/ASparse computation

    Note: Sparse attention support is primarily targeted at NVIDIA Hopper and newer architectures.

  3. Overview of CBTS Rules

    main

    The CBTS (Continuous Build and Test System) uses a rule-based architecture to narrow down which test stages need to run based on a code diff. Each rule inherits from a base Rule class and is responsible for identifying specific file patterns and determining the resulting test scope.

    Rule Summary Table

    Rule ClassScopeTriggers on (File Patterns)
    WaivesRulewaiveonlytests/integration/test_lists/waives.txt
    TestsDefRuletestdefonlytests/**/* (any file under tests/)
    TestListRuletestlistonlytests/integration/test_lists/test-db/*.yml
    AutoDeployRuleautodeployonlyexamples/auto_deploy/**, tensorrt_llm/_torch/auto_deploy/** (excl. .md)
    VisualGenRulevisualgenonlyexamples/visual_gen/**, scripts/visualgen_eval/**, tensorrt_llm/_torch/visual_gen/**, tensorrt_llm/media/**, tensorrt_llm/visual_gen/** (excl. .md)
    SpecDecRulespecdeconlytensorrt_llm/_torch/speculative/**, tensorrt_llm/models/{eagle,medusa,redrafter}/**, examples/{eagle,medusa,redrafter,draft_target_model,ngram}/**, examples/llm-api/llm_speculative_decoding.py (excl. .md)
    AgentFlowRuleagentflowonlyagent-flow/** (excl. .md)
    OutOfScopeRulenooptests/integration/test_lists/{qa,dev}/**, tests/integration/defs/.test_durations*, tests/microbenchmarks/**, **/*.md

  4. Introduction to the LLM API

    main
    The LLM API is a high-level Python API designed to streamline LLM inference workflows. It supports various use cases ranging from single-GPU setups to multi-GPU and multi-node deployments, featuring built-in support for parallelism strategies and advanced features. The API uses a PyTorch-native and modular backend, allowing for customization and integration with the NVIDIA Dynamo ecosystem.

  5. Overview of CBTS Layer C Coverage Utils

    main

    CBTS Layer C is a CI tooling system designed to capture per-test function/class-level coverage. It tracks which product functions each test enters, including coverage from subprocesses, specifically on single-GPU L0 stages.

    Key characteristics:

    • Low Overhead: Uses sys.monitoring (PY_START) in Python 3.12+, meaning overhead scales with the number of functions entered rather than lines executed.
    • CI-Only: This is infrastructure tooling; it does not ship in the product wheel. Every file is a no-op unless the CBTS_COVERAGE_CONFIG environment variable is set.
    • Granularity:
      • Integration tests: Each pytest item is treated as its own context.
      • Unit tests: A batch of tests runs under a single inherited CBTS_TEST_ID context.
      • Hierarchy: Results roll up from function $\rightarrow$ class $\rightarrow$ file using co_qualname (e.g., Class.method).

  6. What is TensorRT LLM?

    main

    TensorRT LLM is an open-source library designed to optimize inference for Large Language Models (LLMs) and Visual Generation models on NVIDIA GPUs. It provides state-of-the-art optimizations including:

    • Custom Kernels: Optimized operations for attention, GEMMs, MoE, and more.
    • Algorithmic Runtime Optimizations: Features like Prefill-Decode disaggregation, Wide Expert Parallelism, and Speculative Decoding.
    • High-Level Python LLM API: Supports various deployment scales from single-GPU to multi-GPU or multi-node setups.
    • Modular Architecture: Built on PyTorch, allowing developers to extend functionality or customize pre-defined models using native PyTorch code.

    The library integrates with the broader NVIDIA inference ecosystem, including NVIDIA Dynamo and Triton Inference Server.

  7. Overview of TensorRT LLM

    main

    TensorRT LLM is an open-source library designed to optimize inference for Large Language Models (LLMs) and Visual Generation models. It achieves high performance through:

    • Specialized Kernels: Optimized operations specifically for common LLM workloads.
    • Efficient Runtime: A high-performance execution engine.
    • Pythonic Framework: A flexible interface that allows developers to customize and extend the system.

    Supported Environments & Dependencies

    • Python: 3.10, 3.12
    • CUDA: 13.2.1
    • PyTorch: 2.11.0
    • License: Apache 2.0

  8. Overview of trtllm-eval

    main

    The trtllm-eval command is a unified entry point for accuracy evaluation, designed for developers to verify and debug model accuracy. It is built on the offline LLM API, which provides clearer error messages and a simpler debugging workflow compared to the online trtllm-serve API.

    Note: trtllm-eval is intended for development and testing purposes and is not recommended for production use.

  9. Overview of the Time Breakdown Tool

    main

    The Time Breakdown tool is a standalone utility designed to analyze and visualize performance metrics from TensorRT-LLM servers. It helps developers understand how time is spent processing individual requests by generating interactive visualizations.

    The tool supports both aggregated and disaggregated server configurations and produces the following outputs:

    1. Interactive HTML Diagram: A stacked bar chart representing the timing breakdown per request, featuring hover tooltips for detailed inspection.
    2. Per-step CPU and GPU timing: A granular breakdown of each generation step.
    3. Statistics: Optional calculation of median times for each timing segment.

  10. Overview of Perf Sanity Scripts

    main

    The jenkins/scripts/perf/ directory provides a suite of tools for running performance sanity tests and managing performance data within the TensorRT-LLM repository. It includes scripts for local and CI-based job submission to SLURM clusters, configuration management, and utilities for regression detection and reporting.

    Key Components:

    • Submit Scripts: local/submit.py (for local aggregated/disaggregated runs) and disaggregated/submit.py (for CI pipelines).
    • Local Execution Wrapper: local/run_disagg.sh provides a config-driven way to submit multiple tests via SLURM.
    • Utilities: perf_utils.py handles baseline computation, regression detection, and chart generation; get_pre_merge_html.py and perf_sanity_triage.py manage reporting and notifications.

  11. Overview of the TensorRT-LLM Lightweight RPC

    main
    The tensorrt_llm.executor.rpc module provides a pure-Python lightweight Remote Procedure Call (RPC) framework designed to simplify IPC (Inter-Process Communication) and TCP-based communication. It supports multiple execution modes including synchronous, asynchronous, future-based, and streaming calls.

  12. Overview of CBTS (Change-Based Testing Selection)

    main

    CBTS is a CI test-selection tool designed to narrow down Jenkins stages and per-stage tests based on the specific files changed in a Pull Request (PR). By identifying only the affected components, it reduces CI execution time.

    Key characteristics:

    • It only subtracts tests; it never adds more than the baseline.
    • If it cannot determine a specific narrowing, it falls back to the existing baseline filter chain.
    • New rules for test selection are implemented in Python only.
    • It operates across three consumption layers: Stage selection (Layer 2), Split-resize (Layer 2.5), and Within-stage test narrowing (Layer 3).