NVIDIA DeepStream TAO Apps

The MDX perception sample application implements two distinct DeepStream pipelines using TAO 4.0 pretrained models:

1. Retail Item Recognition: Detects retail items from video and extracts embedding vectors from every detection bounding box. These embeddings can be used to query a database for the closest match.
2. People ReID: Detects people from video and extracts embedding vectors from every detection bounding box.

Both pipelines utilize a primary GIE (General Inference Engine) module for object detection and a secondary GIE module for embedding extraction.

The DeepStream LPR (License Plate Recognition) sample application demonstrates a multi-stage pipeline for car license plate recognition using TAO 3.0 models. The pipeline follows a hierarchical detection and classification structure:

PGIE(car detection) -> SGIE(car license plate detection) -> SGIE(car license plate recognization)

It utilizes three specific TAO models:

• Car detection model: nvidia:tao:trafficcamnet
• LPD (car license plate detection) model: nvidia:tao:lpdnet
• LPR (car license plate recognization/text extraction) model: nvidia:tao:lprnet

Note: trafficcamnet and LPD models use INT8 precision, while the LPR model uses FP16 precision.

This project provides sample applications to run twelve different NVIDIA TAO models (including Peoplenet Transformer, CitySemSegFormer, Mask2Former, and more) using the DeepStream SDK.

The repository structure includes:

• apps: Sample applications for detection and segmentation models.
• configs: DeepStream nvinfer configuration files and label files.
• post_processor: Inference postprocessors for the models.
• models: Sample model files.
• TRT-OSS: Instructions for building/downloading OSS nvinfer plugins required for certain models with DeepStream 7.1 GA.

Pipeline Architecture: uridecoderbin $\rightarrow$ streammux $\rightarrow$ nvinfer (detection) $\rightarrow$ nvosd $\rightarrow$ (display | encode $\rightarrow$ filesink | encode $\rightarrow$ fakesink)

To switch between different models (e.g., switching from peoplenet to peoplenet_transformer_v2) or different inference engines (e.g., nvinfer vs nvinferserver), you must modify the [primary-gie] section within the configuration file.

Key parameters in the [primary-gie] section:

• plugin-type: Set to 0 for nvinfer or 1 for nvinferserver.
• config-file: The path to the specific model configuration file. Comment out the current file and uncomment the desired one to switch models.

Example configuration snippet:

[primary-gie]
enable=1
# (0): nvinfer; (1): nvinferserver
plugin-type=0
gpu-id=0
batch-size=1
gie-unique-id=1
config-file=nvinfer/config_infer_primary_peoplenet.txt
#config-file=nvinfer/config_infer_primary_peoplenet_transformer_v2.txt
#config-file=triton/config_infer_primary_peoplenet.txt

From DeepStream 6.1 onwards, the LPR application supports three inferencing modes:

1. gst-nvinfer: Based on TensorRT (default). No additional Triton setup required.
2. gst-nvinferserver (Triton CAPI client): Only for x86 platforms. Requires Triton Inference Server libraries.
3. gst-nvinferserver (Triton gRPC client): Only for x86 platforms. Requires Triton Inference Server libraries.

If using nvinferserver (CAPI or gRPC) on x86, you must install the Triton Inference Server libraries or use the DeepStream Triton container. For specific setup instructions, refer to triton_server.md (for CAPI) or triton_server_grpc.md (for gRPC) within the repository.

You can configure the application to export metadata to a Kafka broker by modifying configs/app/retail_object_detection_recognition.yml to enable sink2. This allows the DeepStream pipeline to publish detection and embedding data to Kafka, which can then be consumed by external services.

To test this, you can run a Kafka broker using Docker Compose and a Python consumer script.

The sample applications support three inferencing methods: TensorRT-based gst-nvinfer, Native Triton Inference Server, and Stand-alone Triton gRPC server.

To use Triton or Triton gRPC, you must use YAML configurations and modify the primary-gie section in your app config (e.g., configs/app/det_app_config.yml).

Set plugin-type: 1 (which corresponds to nvinfeserver) and update the config-file-path to point to the appropriate Triton configuration file.

primary-gie:
  # 0: nvinfer, 1: nvinfeserver
  plugin-type: 1

  # Example for peoplenet-transformer using Triton
  config-file-path: ../triton/peoplenet_transformer_tao/pgie_peoplenet_transformer_tao_config.yml

If you encounter LFS (Large File Storage) issues when downloading, you can download the pre-built libnvinfer_plugin.so.8.6.2 directly using wget. This specific binary was built for Jetson Xavier using Jetpack GA (CUDA-12.2, cuDNN v8.6, TensorRT 8.6.1.2).

wget https://nvidia.box.com/shared/static/1ih810ui4z52nvwznfk1xaxvdej3mauw -O libnvinfer_plugin.so.8.6.2

If you encounter LFS (Large File Storage) issues when downloading the repository, you can download the pre-built libnvinfer_plugin.so.7.0.0.1 directly using wget.

Note: The provided libnvinfer_plugin.so.7.1.3 was built specifically for:

• Jetson NX
• Jetpack 4.4GA (CUDA-10.2, cuDNN v8.0, TensorRT 7.1.3)

wget https://nvidia.box.com/shared/static/ezrjriq08q8fy8tvqcswgi0u6yn0bomg.1 -O libnvinfer_plugin.so.7.0.0.1

Run the tritonserver command to start the service. If the server is running on a remote machine, you must update the gRPC URL in all configuration files located in deepstream_app_tao_configs/triton-grpc to point to the server's IP address.

tritonserver --model-repository=/samples/models --strict-model-config=false --grpc-infer-allocation-pool-size=16 --log-verbose=1 --exit-on-error=false

Before using the Pyservicemaker applications, ensure DeepStream SDK 9.0 GA is installed and verified (e.g., by running deepstream-test1). You must install the Pyservicemaker wheel and additional audio/video packages using the DeepStream installation scripts. Additionally, Eigen development packages must be installed and symlinked.

# install pyservicemaker wheel
/opt/nvidia/deepstream/deepstream/install.sh

# install deepstream dependencies
/opt/nvidia/deepstream/deepstream/user_additional_install.sh

# install Eigen development packages
sudo apt install libeigen3-dev
cd /usr/include
sudo ln -sf eigen3/Eigen Eigen