DOCSEARCHbyAlgolia
Submit Docs

Triton Inference Server Documentation

repository·main·Indexed 27 days ago

https://github.com/triton-inference-server/server

Open-source inference serving software for deploying AI models from frameworks including TensorRT, PyTorch, ONNX, OpenVINO, Python, and RAPIDS FIL. Supports concurrent model execution, dynamic batching, and sequence batching across cloud, data center, edge, and embedded devices. Features HTTP/REST and gRPC protocols based on KServe, a Backend API for custom extensions, and an In-Process Server API for C, C++, Java, and Python integration.

Tokens
68.8K
Snippets
135
Records
322
Agent score
94%

What's inside Triton Inference Server

  1. Overview of Triton Model Analyzer

    main
    Triton Model Analyzer is a tool designed to characterize GPU memory and compute utilization by sending requests to your models using Performance Analyzer. It is primarily used to determine GPU memory requirements under various batching and model instance configurations, enabling intelligent decisions on how to co-locate multiple models on a single GPU without exceeding memory capacity.

  2. Overview of Triton Inference Server

    main

    Triton Inference Server is an open-source inference serving software designed to streamline AI inferencing. It allows teams to deploy models from various deep learning and machine learning frameworks (including TensorRT, PyTorch, ONNX, OpenVINO, Python, and RAPIDS FIL) across cloud, data center, edge, and embedded devices.

    Key capabilities include:

    • Multi-framework support: Deploy models from diverse DL/ML frameworks.
    • Optimized execution: Supports concurrent model execution, dynamic batching, and sequence batching for stateful models.
    • Flexible pipelines: Use Ensembling or Business Logic Scripting (BLS) for model pipelines.
    • Multiple protocols: Supports HTTP/REST and GRPC inference protocols based on the KServe protocol.
    • Extensibility: Provides a Backend API for custom backends (including Python-based backends) and offers C and Java APIs for in-process integration.
    • Observability: Provides metrics for GPU utilization, throughput, and latency.

  4. Use the Sequence Batcher for Stateful Models

    main

    The Sequence Batcher is designed for stateful models where a sequence of inference requests must be routed to the same model instance. Like the dynamic batcher, it combines non-batched requests into batches dynamically.

    Configuration is handled via the ModelSequenceBatching property in the model configuration, which controls sequence timeouts and control signals (sequence start, end, ready, and correlation ID).

  5. Communicate with Triton using HTTP/REST and GRPC

    main

    Triton supports communication via HTTP/REST and GRPC protocols based on KServe standards, along with Triton-specific extensions.

    GRPC Streaming: Triton provides a bi-directional streaming version of the inference RPC. Use streaming if:

    • You need to ensure a sequence of requests hits the same Triton instance behind a Load Balancer (by holding a single connection).
    • You need to preserve the order of requests/responses over the network.

    For most standard inference requests, the unary version is recommended.

  6. Understand Triton Inference Server Architecture

    main

    Triton Inference Server operates by serving models from a file-system based model repository. The workflow for an inference request is as follows:

    1. Request Arrival: Requests enter the server via HTTP/REST, gRPC, or the C API.
    2. Routing: Requests are routed to a specific per-model scheduler.
    3. Scheduling & Batching: The scheduler applies configured scheduling and batching algorithms (which can be set on a per-model basis).
    4. Backend Execution: The scheduler passes batched requests to the appropriate backend (e.g., a specific deep-learning framework or a custom backend). The backend performs the actual inference.
    5. Response: The backend produces outputs which are returned to the client.

    Additionally, Triton provides a model management API (via HTTP/REST, gRPC, or C API) to query and control models, and exposes health endpoints (readiness/liveness) and metrics (utilization, throughput, latency) for deployment integration (e.g., Kubernetes).

  7. Caching in Ensemble Models

    main

    Ensemble models support caching under the following conditions:

    • Top-level Caching: An ensemble model supports caching only if all composing models within the ensemble also have response caching enabled.
    • Cache Hit Behavior: A cache hit at the ensemble level skips all composing models and returns the cached result immediately.
    • Cache Miss Behavior: A cache miss at the ensemble level falls back to standard inference, proceeding to the composing models.
    • Granularity: The ensemble and its composing models can have independent caching configurations. It is possible for an ensemble to miss its cache but for an intermediate model within the ensemble to hit its own cache.

  8. Use the In-Process Triton Server API

    main

    The Triton Inference Server provides a backwards-compatible C API, along with Python and Java bindings, that allows you to link Triton directly into your C, C++, Java, or Python application. This is known as the "In-Process Triton Server API" or "Server API".

    All Triton capabilities are encapsulated in the shared library and exposed via this API. While the standard tritonserver executable uses this API to implement HTTP/REST and gRPC endpoints, you can use it directly in your own application to embed Triton functionality without the overhead of network communication.

  9. Quickstart: Run TRT-LLM models with Triton Server via LLM API

    main

    You can serve HuggingFace models directly using the TensorRT-LLM backend and the PyTorch LLM API without requiring engine compilation. Follow these steps to set up a running server:

    1. Launch the container: Use the specialized trtllm-python-py3 container image.
    2. Clone TensorRT-LLM: Clone the official repository to access the backend scripts and model configurations.
    3. Configure the model: Edit the model.yaml file to specify your HuggingFace model ID or local path. All keys in model.yaml map to LLM() constructor arguments (e.g., for KV cache, quantization, or parallelism).
    4. Launch the server: Run the launch_triton_server.py script from the parent directory of the cloned TensorRT-LLM folder.
    5. Test the endpoint: Send a POST request to the /v2/models/tensorrt_llm/generate endpoint.

  10. Quickstart: Serve a Model in 3 Easy Steps

    main

    Follow these steps to set up a model repository, launch the Triton server using a Docker container, and send an inference request using the SDK container.

    Prerequisites:

    • Docker installed with NVIDIA Container Toolkit (for GPU support).
    • Access to NVIDIA GPU Cloud (NGC) containers.

    Steps:

    1. Prepare the model repository: Clone the server repository and run the model fetching script.
    2. Launch Triton: Run the tritonserver from the NGC container, pointing it to your model repository.
    3. Send an Inference Request: Use the image_client from the Triton SDK container to query the running server.
    # Step 1: Create the example model repository
git clone -b r26.06 https://github.com/triton-inference-server/server.git
cd server/docs/examples
./fetch_models.sh

# Step 2: Launch triton from the NGC Triton container
docker run --gpus=1 --rm --net=host -v ${PWD}/model_repository:/models nvcr.io/nvidia/tritonserver:26.06-py3 tritonserver --model-repository=/models --model-control-mode explicit --load-model densenet_onnx

# Step 3: Sending an Inference Request
# In a separate console, launch the image_client example from the NGC Triton SDK container
docker run -it --rm --net=host nvcr.io/nvidia/tritonserver:26.06-py3-sdk /workspace/install/bin/image_client -m densenet_onnx -c 3 -s INCEPTION /workspace/images/mug.jpg

  12. Verify model compatibility and deployment

    main

    Before deploying, ensure your model is compatible with a supported Triton backend.

    • For ONNXRuntime and TensorRT backends: You can use Triton's AutoComplete feature to infer minimal model configuration. A config.pbtxt is optional unless you need to set specific parameters. To see the complete configuration Triton generates internally, start the server with the --log-verbose=1 flag.
    • For other backends: Refer to the Minimal Model Configuration requirements for that specific backend.
    • For unsupported models: Use the Python Backend for a simple implementation (though it may be less performant) or write a Custom C++ Backend for maximum performance.