TensorRT Edge-LLM Documentation

The NvFP4MoEPluginGeforce is a TensorRT plugin designed for SM12x (consumer Blackwell) architectures. It wraps the fused NVFP4 MoE kernel from the CuTeDSL library.

It is the counterpart to Nvfp4MoePlugin (which targets SM110/Thor). While they share the same ONNX input/output surface, they require different weight layouts:

• NvFP4MoEPluginGeforce (SM12x): Expects FC1 packed as a plain [up_all, gate_all] concatenation along the M axis.
• Nvfp4MoePlugin (SM110): Expects FC1 packed with a 64-row up/gate interleave [up_chunk(64), gate_chunk(64), ...].

Python export logic automatically selects the correct plugin and weight repack based on the target architecture.

TensorRT Edge-LLM is a high-performance C++ inference runtime designed for Large Language Models (LLMs) and Vision-Language Models (VLMs) on NVIDIA edge platforms. It is optimized for resource-constrained devices including:

• NVIDIA Jetson
• NVIDIA DRIVE
• NVIDIA DGX Spark

Key capabilities include:

• Efficient Deployment: Enables state-of-the-art model execution on embedded hardware.
• Checkpoint Conversion: Provides Python scripts to convert HuggingFace checkpoints to ONNX format.
• Edge-Native Workflow: Both engine building and end-to-end inference runs entirely on the edge platform itself.

The Nvfp4MoePlugin is a TensorRT plugin designed for SM100, SM101, and SM110 architectures. It wraps the decomposed split FC1/FC2 NVFP4 MoE kernel located in kernelSrcs/nvfp4_moe_cutedsl/.

It is the architectural counterpart to NvFP4MoEPluginGeforce (which targets SM120/SM121 consumer Blackwell). While both plugins share the same ONNX input surface and attribute set, they require different on-disk weight layouts:

• Nvfp4MoePlugin (this plugin): Expects FC1 packed as a 64-row up/gate interleave: [up_chunk(64), gate_chunk(64), up_chunk(64), ...] along the M axis.
• NvFP4MoEPluginGeforce: Expects FC1 packed as a plain [up_all, gate_all] concatenation along the M axis.

The Python export process automatically selects the correct plugin and weight repack based on the target architecture.

TensorRT Edge-LLM provides several end-to-end workflow examples covering various multimodal and specialized inference use cases. These examples demonstrate how to deploy and optimize different model architectures using the library's capabilities.

Available workflows include:
- VLM (Vision-Language Model)
- Speculative Decoding (EAGLE3, MTP, DFlash)
- Phi-4 Multimodal (with LoRA merge)
- ASR (Automatic Speech Recognition) with Qwen3-ASR
- MoE (Mixture of Experts)
- TTS (Text-to-Speech)
- Alpamayo-R1-10B (VLA - Vision-Language-Action)
- Omni (Audio + Vision + Speech I/O)
- Experimental High-Level Python API and Server

The TensorRT Edge-LLM Engine Builder is a C++ component that serves as the second stage in the deployment workflow. It converts ONNX models into optimized TensorRT engines specifically designed for edge deployment.

Key Responsibilities:

• Parsing ONNX models and extracting network structure.
• Configuring optimization profiles for dynamic shapes.
• Compiling TensorRT engines with platform-specific optimizations.
• Generating runtime configuration files.
• Handling model-specific requirements such as speculative decoding, VLM, and LoRA.

⚠️ VERSION COMPATIBILITY WARNING: ONNX models and TensorRT engines are NOT portable across different versions of TensorRT Edge-LLM or TensorRT. You must always re-export ONNX models and rebuild engines when upgrading versions.

The kernel is highly specialized for the Gemma 4 audio configuration using constexpr values. It is optimized for the following parameters:

• chunkSize: 12
• leftHorizon: 12
• contextSize: 24
• relPosLen: 13
• headDim: 128

Generality Warning: Because these shapes are hard-coded as constexpr, the kernel is not generic. To support different configurations, you must either generalize the constexpr block shapes and shared-memory layouts or use a reference path.

The server supports OpenAI-compatible tool calling. When a model returns a tool call, you must include the assistant's tool_calls and the subsequent tool response in your next request to continue the loop.

Example message sequence for a tool follow-up:

{
  "messages": [
    {"role": "user", "content": "What is the weather in Paris?"},
    {
      "role": "assistant",
      "content": null,
      "tool_calls": [{
        "id": "call_1",
        "type": "function",
        "function": {
          "name": "get_weather",
          "arguments": "{\"city\":\"Paris\"}"
        }
      }]
    },
    {
      "role": "tool",
      "tool_call_id": "call_1",
      "content": "{\"temperature\":22,\"unit\":\"celsius\"}"
    }
  ],
  "tools": [{
    "type": "function",
    "function": {"name": "get_weather", "parameters": {"type": "object"}}
  }]
}

The exporter follows a strict checkpoint-driven pipeline:

1. Configuration: Read checkpoint configuration and the safetensors index.
2. Promotion: For multimodal models, promote nested LLM configs if the root config is a wrapper.
3. Config Building: Construct a ModelConfig containing quantization, KV cache, layer-type, RoPE, and model-family details.
4. Dispatch: Select the appropriate TensorRT Edge-LLM-native model implementation based on the config.
5. Weight Loading: Load and repack weights from the checkpoint tensors.
6. Vocab Reduction (Optional): Apply vocabulary reduction using vocab_map.safetensors.
7. ONNX Export: Export graphs using custom-op schemas and dynamo translations.
8. Sidecar Generation: Write runtime sidecars (tokenizer, config, etc.) for consumption by the engine builder and runtime.

Speculative decoding is used to increase generation throughput by using a smaller draft model to predict tokens. Two main types are supported:

1. EAGLE3 Speculative Decoding: Requires an external draft model (e.g., Qwen3-1.7B_eagle3). It provides significant speedup, especially at Batch Size 1 (latency-bound scenarios).
2. MTP (Multi-Token Prediction) Speculative Decoding: Uses the model's built-in draft heads. No external draft checkpoint is required. In v0.7.1, MTP significantly improved Qwen3.5 VLM throughput (1.21x to 2.12x over vanilla decoding).

Key Metrics:

• Accept Rate: The percentage of tokens predicted by the draft model that are accepted by the base model.
• Speedup: The factor by which generation throughput increases compared to vanilla decoding.

For setup instructions, see the Speculative Decoding guide.