DOCSEARCHbyAlgolia
Submit Docs

NVIDIA TensorRT

repository·main·Indexed 11 days ago

https://github.com/nvidia/tensorrt

An NVIDIA SDK for high-performance deep learning inference. This repository provides open-source components including plugins, the ONNX parser, and sample applications. It includes specialized optimizations for DeBERTa model inference via the disentangledAttentionPlugin (starting from TensorRT 8.4.3) and accelerated implementations for Stable Diffusion, Flux, and Cosmos model families.

Tokens
156.1K
Snippets
424
Records
723
Agent score
95%

What's inside TensorRT

  1. Overview of embLayerNormPlugin

    main

    The embLayerNormPlugin is a TensorRT plugin designed to perform two primary tasks in a single operation:

    1. Embedding Lookup and Normalization: It embeds an input sequence by performing token embedding lookup, segment embedding lookup, adding positional embeddings, and finally applying layer normalization.
    2. Mask Preprocessing: Depending on the version, it can preprocess input masks used to identify valid tokens in padded sequences.

    Versioning Recommendations

    • Deprecated: Versions 1-3 (using IPluginV2DynamicExt) are deprecated and will be removed.
    • Recommended: Versions 4-6 (using IPluginV3) should be used for new implementations.

    Hardware Requirements

    • This plugin requires GPUs with compute capability >= 7.0.

  2. Overview of Polygraphy capabilities

    main

    Polygraphy is a toolkit for deep learning inference prototyping and debugging. Key capabilities include:

    • Framework Comparison: Run inference across multiple backends (e.g., TensorRT and ONNX-Runtime) and compare results.
    • Model Conversion: Convert models to various formats, such as creating TensorRT engines with post-training quantization (INT8 calibration).
    • Model Inspection: View detailed information about different model types.
    • ONNX Manipulation: Use the surgeon tools via CLI to extract subgraphs, simplify, or sanitize ONNX models (e.g., folding constants).
    • TensorRT Debugging: Isolate faulty tactics in TensorRT to identify issues in the optimization process.

  3. Overview of DeBERTa Model Inference with TensorRT Optimizations

    main

    This demo provides a workflow for optimizing DeBERTa model inference using a specialized TensorRT plugin designed for 'disentangled attention'.

    Why use this optimization?

    DeBERTa's disentangled attention design triples the attention computation compared to standard BERT and involves indirect memory accesses during gather operations. To mitigate this, a disentangledAttentionPlugin is used to optimize these modules. This plugin is built-in starting from TensorRT 8.4.3.

    Workflow Summary

    The demo covers the following pipeline:

    1. Export: Convert a PyTorch DeBERTa model (using HuggingFace implementation) to ONNX format.
    2. Modify: Modify the ONNX model by inserting the disentangledAttentionPlugin nodes.
    3. Inference: Run inference using one of three methods:
      • TensorRT trtexec executable.
      • TensorRT Python API.
      • ONNX Runtime with the TensorRT Execution Provider.
    4. Validation: Measure performance benchmarks and perform correctness checks to compare models with and without the plugin.

  4. Overview of the trtexec command-line tool

    main

    The trtexec tool is a command-line wrapper included in the samples directory that allows you to utilize TensorRT without developing a custom application. It serves two primary purposes:

    1. Benchmarking networks: Test the performance of running inference on a model (e.g., an ONNX file) using random or user-provided input data. It supports various options for specifying inputs, outputs, iterations, and precision.
    2. Serialized engine generation: Generate serialized engine files that can be loaded into other applications for inference. Note that in INT8 mode, random weights are used during this process.

  5. Overview of the Polygraphy comparator module

    main

    The polygraphy.comparator module provides tools for comparing results between different models or execution environments (e.g., comparing a TensorRT engine against a PyTorch model). The module is organized into several key components:

    • comparator: The core logic for orchestrating comparisons.
    • data_structures: Definitions for how comparison results and data are represented.
    • compare_func: Functions used to perform the actual comparison logic.
    • postprocess_func: Functions for processing or cleaning up results after comparison.
    • data_loader: Utilities for loading the data required for comparison.

  6. Overview of PyTorch-Quantization toolkit

    main

    PyTorch-Quantization is a toolkit designed for training and evaluating PyTorch models using simulated quantization.

    Key Features:

    • Automatic or Manual Quantization: Quantization can be added to models automatically or manually to tune for accuracy and performance.
    • Hardware Acceleration: Compatible with NVIDIA's high-performance integer kernels that leverage integer Tensor Cores.
    • TensorRT Integration: Quantized models can be exported to ONNX format and subsequently imported by TensorRT 8.0 and later versions.

  7. Overview of Polygraphy

    main

    Polygraphy is a Python toolkit designed to assist in running and debugging deep learning models across various frameworks. It provides both a Python API and CLI tools for model verification and debugging workflows.

    Important Note on API Usage: Only APIs explicitly documented in the Polygraphy documentation should be considered public. Any undocumented APIs are considered internal, do not adhere to the deprecation policy, and may be modified or removed without warning. Avoid using undocumented APIs in production or stable workflows.

  8. Overview of FFTPlugin

    main

    The FFTPlugin computes Fast Fourier Transforms (FFT) using cuFFT. It transforms the trailing ndims dimensions of the input and batches over the leading dimensions.

    Supported Transform Types:

    • C2C (Complex-to-Complex)
    • R2C (Real-to-Complex, onesided)
    • C2R (Complex-to-Real, onesided)

    Supported Precisions: FP32, FP16, and BF16 for 1D, 2D, and 3D signals.

    Note on Usage: This plugin backs the ONNX DFT operator. Most users do not need to construct the plugin directly; the TensorRT ONNX parser automatically routes DFT nodes to this plugin.

  9. View TensorFlow Quantization Model Zoo Results

    main

    The Model Zoo provides performance benchmarks for various models using Post-Training Quantization (PTQ) and Quantization-Aware Training (QAT) with TensorRT. These results demonstrate the accuracy and latency trade-offs when moving from a TensorFlow baseline to a quantized TensorRT implementation.

    Benchmark Environment:

    • GPU: NVIDIA A100
    • TensorRT Version: 8.4
    • Accuracy Metric: Top-1 validation accuracy on the full ImageNet dataset.

    Key Concepts:

    • Baseline (TensorFlow): The original unquantized model performance.
    • PTQ (TensorRT): Post-Training Quantization results.
    • QAT (TensorRT): Quantization-Aware Training results, typically offering higher accuracy than PTQ at the cost of a training step.

    For specific implementation examples, refer to the following directories:

  11. Understand NVIDIA TensorFlow 2.x Quantization features and dependencies

    main

    This toolkit quantizes TensorFlow 2.x Keras models for Quantization-Aware Training (QAT) by inserting Quantization and DeQuantization (QDQ) nodes. It follows NVIDIA's QAT recipe to optimize acceleration with TensorRT on NVIDIA GPUs.

    Key Features

    • Automated or manual insertion of QDQ nodes.
    • Support for adding new layer types.
    • Programmatic control over quantization behavior.
    • Automatic tests for residual and inception architecture blocks.
    • Utilities for converting TensorFlow 2.x models to TensorRT via ONNX.

    Dependencies

    • Python >= 3.8
    • TensorFlow >= 2.8
    • tf2onnx >= 1.10.1
    • onnx-graphsurgeon
    • pytest
    • pytest-html
    • TensorRT (optional) >= 8.4 GA

  12. Substitute ONNX subgraphs with plugins using Polygraphy plugin tool

    main

    The polygraphy plugin tool automates the process of replacing specific subgraphs within an ONNX model with custom plugins.

    To use this tool, your plugins must define the graph patterns they are capable of substituting. This is achieved by including a pattern.py file within the plugin's directory.

    Workflow:

    1. Use match to identify potential substitution opportunities and generate a config.yaml file.
    2. Edit the generated config.yaml to specify exactly which plugins should replace which subgraphs.
    3. Use replace to perform the actual substitution in the model based on your configuration.