DOCSEARCHbyAlgolia
Submit Docs

Triton Inference Server Client Libraries

repository·main·Indexed 20 days ago

https://github.com/triton-inference-server/client

Client libraries and examples for communicating with the Triton Inference Server via HTTP/REST and gRPC protocols. Provides APIs for C++, Python, and Java, supporting features such as inferencing, model repository management, system and CUDA shared memory for high-performance I/O, and SSL/TLS configuration.

Tokens
26.4K
Snippets
84
Records
121
Agent score
72%

What's inside triton-inference-server-client

  1. Overview of Triton Client Libraries

    main

    Triton provides several client libraries to simplify communication with the Triton Inference Server. These libraries allow you to access capabilities such as inferencing, status and health, statistics and metrics, and model repository management.

    Supported Languages and Protocols

    • C++ and Python APIs: Supports both HTTP/REST and GRPC protocols. These libraries also support using system and CUDA shared memory for high-performance input/output passing.
    • Java API: Supports HTTP/REST requests (currently a limited feature subset).
    • gRPC via protoc: You can use the protoc compiler to generate gRPC APIs for many other languages. Examples are provided for Go, Java/Scala, and JavaScript.

    Example Applications

    • image_client: C++ and Python versions for executing image classification models.
    • Simple C++/Python Examples: Demonstrates HTTP/REST (prefixed with simple_http_) and gRPC (prefixed with simple_grpc_) usage.
    • Java Examples: Demonstrates basic inferencing tasks.
    • Python gRPC Examples: Uses generated gRPC client stubs (e.g., grpc_client.py and grpc_image_client.py).

  2. Use CUDA Shared Memory for Performance

    main

    CUDA shared memory allows for high-performance tensor communication between the client and Triton by avoiding host-to-device copies.

    In Python, the client library provides the tritonclient.utils.cuda_shared_memory module to manage CUDA shared memory. This module supports:

    • numpy arrays
    • DLPack tensors

  3. Manage Sequences for Stateful Models

    main

    When performing inference with stateful models, the client is responsible for managing the lifecycle of a sequence.

    To correctly handle stateful inference, you must:

    1. Assign a Sequence ID: Generate and provide a unique sequence ID for each sequence.
    2. Mark Sequence Start: Set the appropriate flag on the first inference request of a sequence.
    3. Mark Sequence End: Set the appropriate flag on the last inference request of a sequence.

  4. Use System Shared Memory for Performance

    main

    System shared memory can be used to communicate tensors between the client library and Triton to significantly improve performance.

    In Python, since there is no standard way to allocate and access shared memory, the client library provides a utility module: tritonclient.utils.shared_memory. This module can be used to create, set, and destroy system shared memory.

  5. Use Ensemble Models for Preprocessing

    main
    Ensemble models allow you to send raw data (like raw image binaries) directly to Triton, where a pipeline of models (e.g., a DALI backend for preprocessing followed by a TensorFlow/PyTorch model) handles the transformation. This avoids the need to perform expensive preprocessing on the client side.

  6. Handle BYTES and String Datatypes in Python

    main

    When working with tensors containing variable-length binary data (the BYTES datatype), the Python client library uses numpy for representation.

    To ensure correctness, use np.object_ as the dtype for your numpy arrays.

    Warning: Avoid using np.bytes_ for BYTES tensors. While supported for backwards compatibility, np.bytes_ causes numpy to strip trailing zeros from array elements, which will corrupt binary sequences that end in zero(s).

    # Recommended way to represent BYTES tensors
import numpy as np
input_data = np.array(['string_data', b'binary_data'], dtype=np.object_)

  7. Retrieve ORCA header metrics for LLM KV-cache

    main

    If using the TensorRT-LLM backend, Triton can include live KV-cache utilization and capacity metrics in the HTTP response headers.

    To request these metrics, include the endpoint-load-metrics-format header in your HTTP inference request with one of the following values:

    • text: Returns native HTTP, comma-separated key-value pairs (e.g., endpoint-load-metrics: TEXT cpu_utilization=0.3, mem_utilization=0.8).
    • json: Returns a JSON encoding of the metrics (e.g., endpoint-load-metrics: JSON {"cpu_utilization": 0.3, ...}).

  8. Run the Image Classification Example

    main

    The image_client (C++) and image_client.py (Python) applications demonstrate how to send image classification requests to a running Triton server.

    Command Line Arguments

    • -m <model_name>: The name of the model to use (e.g., inception_graphdef).
    • -s <model_instance_name>: The model instance name (e.g., INCEPTION).
    • -i <protocol>: Specify the protocol. Defaults to http. Use grpc to switch to GRPC.
    • -u <url>: The endpoint URL (required if using grpc, e.g., localhost:8001).
    • -c <count>: Number of top classifications to display.
    • -b <batch_size>: The batch size for inferencing. If the batch is larger than the provided images, the client repeats images to fill the batch.

    Examples

    Basic HTTP request for a single image:

    $ image_client -m inception_graphdef -s INCEPTION qa/images/mug.jpg

    GRPC request with specific endpoint and top 3 results:

    $ image_client -i grpc -u localhost:8001 -m inception_graphdef -s INCEPTION -c 3 qa/images/mug.jpg

    Batch request using a directory of images:

    $ image_client -m inception_graphdef -s INCEPTION -c 3 -b 2 qa/images
    $ image_client -m inception_graphdef -s INCEPTION -c 3 -b 2 qa/images/mug.jpg

  9. Install the Python client library via pip

    main

    The Triton client libraries for Python can be installed using pip. You can choose to install the full suite or specific protocol support.

    • Install everything (HTTP, gRPC, and CUDA shared memory support): pip install tritonclient[all]

    • Install only HTTP/REST support: pip install tritonclient[http]

    • Install HTTP/REST and CUDA shared memory support: pip install tritonclient[http, cuda]

    Note: The cuda package must be explicitly specified if you are not using the [all] flag but still require cuda_shared_memory utilities.

    $ pip install tritonclient[all]

  10. Download client libraries from GitHub releases

    main

    You can download pre-built C++, Python, and Java client libraries as a tarball from the Triton GitHub release page. Look for assets named like v2.3.0_ubuntu2004.clients.tar.gz.

    Once extracted, the files are organized as follows:

    • lib/: Libraries
    • include/: Headers
    • python/: Python wheel files
    • java/: JAR files
    • bin/ and python/: Built examples
    $ mkdir clients
$ cd clients
$ wget https://github.com/triton-inference-server/server/releases/download/<tarfile_path>
$ tar xzf <tarfile_name>

  11. Download Triton SDK Docker image from NGC

    main

    An NVIDIA GPU Cloud (NGC) Docker image is available containing the client libraries and examples.

    Command: docker pull nvcr.io/nvidia/tritonserver:<xx.yy>-py3-sdk (replace <xx.yy> with your desired version).

    Internal Paths:

    • Libraries: /workspace/install/lib
    • Headers: /workspace/install/include
    • Python wheels: /workspace/install/python

    Important for CUDA Shared Memory: When using the CUDA shared memory feature with Docker, you must add the --pid host flag when launching your containers. This is required because CUDA IPC APIs need the source and destination PIDs to be different, and Docker's default PID namespace isolation can prevent this.

    $ docker pull nvcr.io/nvidia/tritonserver:<xx.yy>-py3-sdk

  12. Install triton-client in Rust

    main

    To use the triton-client library, add it to your Cargo.toml along with tokio for the async runtime. Note that building the library requires protoc (the Protocol Buffers compiler) to be installed on your system.

    Prerequisites:

    • macOS: brew install protobuf
    • Ubuntu/Debian: apt-get install protobuf-compiler
    [dependencies]
triton-client = { path = "src/rust/triton-client" }
tokio = { version = "1", features = ["full"] }