FasterTransformer Documentation

FasterTransformer provides optimized kernels, memory management, and model parallelism for GPT-style decoding models. It is designed to handle very large models (like GPT-3 with 175B parameters) that cannot fit on a single GPU by supporting multi-GPU and multi-node inference.

Key characteristics of the GPT implementation:

• Uses GeLU activation.
• Does not include an encoder module or cross multi-head attention.
• Optimized for high-efficiency kernels and low latency.

FasterTransformer provides an optimized implementation of the HuggingFace T5 model. It supports multiple checkpoint formats, data types, and inference features designed for high-performance LLM deployment.

Supported Checkpoint Converters

• Huggingface
• Megatron
• NeMo Megatron

Supported Data Types

• FP32
• FP16
• BF16

Key Features

• Multi-GPU multi-node inference
• Dynamic random seed
• Stop tokens
• Beam search and sampling support
• Loading FP32 or FP16 weights

Supported Frameworks

• PyTorch
• Triton backend

FasterTransformer provides highly optimized transformer-based encoder and decoder components for inference. It is built on CUDA, cuBLAS, cuBLASLt, and C++. It automatically utilizes Tensor Cores on Volta, Turing, and Ampere GPUs when using FP16 precision.

Important Notice: FasterTransformer development has transitioned to TensorRT-LLM. Developers are encouraged to use TensorRT-LLM for the latest LLM inference improvements.

FasterTransformer provides the following features for the BART model:

• Checkpoint Loading: Supports Huggingface checkpoints.
• Data Types: FP32, FP16, and BF16.
• Inference Features:
  • Multi-GPU multi-node inference
  • Dynamic random seed
  • Stop tokens
  • Beam search and sampling support
• Framework: PyTorch

Supported Data Types

• FP32, FP16, BF16

Key Features

• Multi-GPU multi-node inference
• Dynamic random seed
• Stop tokens and Bad words list
• Beam search and sampling support
• Checkpoint conversion for EleutherAI and Huggingface

Supported Frameworks

• Triton backend

FasterTransformer provides the following features for the DeBERTa-V2 model:

• Checkpoint Loading: Supports Huggingface models.
• Data Types: FP32, FP16, and BF16.
• Inference: Multi-GPU multi-node inference (implemented, not yet verified).
• Attention Mechanism: Optimized disentangled attention mechanism support with fused kernels.
• Frameworks: PyTorch and TensorFlow.

FasterTransformer can be integrated into various workflows using the following interfaces:

• C API: Direct access to the core implementation.
• TensorFlow/PyTorch OPs: Pre-built operators for use within these frameworks.
• Custom Wrappers: For other frameworks, users can wrap the C++ code to integrate FasterTransformer functionality.

FasterTransformer distinguishes between two main components for transformer-based models:

1. Decoder: Refers specifically to the transformer decoder blocks. Each block contains two attention blocks and a feed-forward network. This is used for lower-level model operations.
2. Decoding: Refers to the entire translation/generation process. This includes position encoding, embedding lookup, multiple decoder layers, and the selection method (beam search or sampling) to choose tokens.

If you require a custom beam search algorithm that is difficult to implement in a custom kernel, it is recommended to use the TensorFlow or PyTorch decoding interfaces with the FasterTransformer Decoder, though this may result in lower performance compared to the native FasterTransformer Decoding implementation, especially at small batch sizes.

To run the Faster Swin-Transformer demo, ensure your environment meets the following requirements:

• CMake: >= 3.13 (required for PyTorch support)
• CUDA: 11.0 or newer
• NCCL: 2.10 or newer
• Python: 3 is recommended
• PyTorch: Verified on 1.10.0; >= 1.5.0 should work

It is recommended to use the Docker image nvcr.io/nvidia/pytorch:22.09-py3.

FasterTransformer provides several scripts to profile BERT performance across different frameworks and precisions. Use the following scripts depending on your target framework and precision:

TensorFlow

• FP32/FP16: benchmarks/bert/tf_benchmark.sh
• INT8: sample/tensorflow/scripts/profile_encoder_performance_int8.sh

PyTorch

• FP32/FP16: sample/pytorch/scripts/profile_encoder.sh
• INT8: sample/pytorch/scripts/profile_encoder_int8.sh