DOCSEARCHbyAlgolia
Submit Docs

DeepStream-Yolo Documentation

repository·master·Indexed 24 days ago

https://github.com/marcoslucianops/deepstream-yolo

Optimized configuration and custom parser implementations for running YOLO models (v5 to v13), RT-DETR, and YOLO-NAS within the NVIDIA DeepStream SDK. Supports x86 and Jetson platforms, providing guides for custom model deployment, CO-DETR ONNX conversion via MMDetection, NMS parameter tuning, and Docker integration.

Tokens
31.1K
Snippets
107
Records
133
Agent score
84%

What's inside DeepStream-Yolo

  1. How to use multiple YOLO GIEs on DeepStream

    master

    The standard deepstream-app does not support multiple primary GIEs (General Inference Engines). To use multiple YOLO models, you must configure one model as the primary GIE and all subsequent models as secondary GIEs. Secondary GIEs perform inference on the objects detected by the primary GIE.

    To implement this, you must:

    1. Set up a directory structure for each GIE.
    2. Compile custom libraries for each GIE folder.
    3. Configure individual config_infer_primary files for each model.
    4. Update the main deepstream_app_config to link the primary and secondary GIEs.

  3. Quickstart: Basic usage of DeepStream-Yolo

    master

    To get started with DeepStream-Yolo, follow these steps to set up a Darknet model (e.g., YOLOv4):

    1. Clone the repository:
      git clone https://github.com/marcoslucianops/DeepStream-Yolo.git
cd DeepStream-Yolo
    2. Download model files: Download the .cfg and .weights files from the Darknet repository and place them in the DeepStream-Yolo folder.
    3. Compile the custom library:
      • Set the CUDA_VER environment variable based on your DeepStream version (see CUDA Version Mapping).
      • Run the make command:
        make -C nvdsinfer_custom_impl_Yolo clean && make -C nvdsinfer_custom_impl_Yolo
    4. Configure the inference file: Edit config_infer_primary.txt to point to your model files:
      [property]
...
custom-network-config=yolov4.cfg
model-file=yolov4.weights
...
      Note: For Darknet models, dynamic batch-size is enabled by default. To use static batch-size, uncomment force-implicit-batch-dim=1 in the config.
    5. Run the application:
      deepstream-app -c deepstream_app_config.txt
      Note: The first run may take over 10 minutes to generate the TensorRT engine.

    If using YOLOv2 or YOLOv2-Tiny, update deepstream_app_config.txt to use config_infer_primary_yoloV2.txt under the [primary-gie] section.

  4. Convert RT-DETR Paddle models to ONNX

    master

    To use RT-DETR Paddle models in DeepStream-Yolo, you must first convert the .pdparams model to ONNX format using the export_rtdetr_paddle.py utility.

    Prerequisites

    1. Clone the RT-DETR repository and install requirements:
    git clone https://github.com/lyuwenyu/RT-DETR.git
cd RT-DETR/rtdetr_paddle
pip3 install -r requirements.txt
pip3 install onnx onnxslim onnxruntime paddle2onnx
    1. Copy export_rtdetr_paddle.py from the DeepStream-Yolo/utils directory into the RT-DETR/rtdetr_paddle folder.
    2. Download your model (e.g., rtdetr_r50vd_6x_coco.pdparams).

    Conversion Command

    Run the export script with the appropriate flags for your DeepStream version:

    • DeepStream >= 6.1: Use --dynamic for dynamic batch-size.
    • DeepStream >= 6.0: Use --simplify to simplify the ONNX model.
    • DeepStream 5.1: Remove --dynamic and use --opset 12 (default is 16).
    • Static Batch Size: Use --batch <size> (e.g., --batch 4).

    After conversion, copy the generated .onnx file and labels.txt to the DeepStream-Yolo folder.

    python3 export_rtdetr_paddle.py -w rtdetr_r50vd_6x_coco.pdparams -c configs/rtdetr/rtdetr_r50vd_6x_coco.yml --dynamic

  6. Convert YOLOX models to ONNX

    master

    To use YOLOX models with DeepStream, you must first convert the .pth weights to an ONNX format using the export_yolox.py utility.

    Prerequisites

    1. Clone the official YOLOX repository and install requirements:
    git clone https://github.com/Megvii-BaseDetection/YOLOX.git
cd YOLOX
pip3 install -r requirements.txt
python3 setup.py develop
pip3 install onnx onnxslim onnxruntime
    1. Copy export_yolox.py from the DeepStream-Yolo/utils directory into the YOLOX folder.

    Conversion Steps

    1. Download your model (e.g., yolox_s.pth).
    2. Run the conversion script. Use the --simplify flag for DeepStream >= 6.0 and --dynamic for dynamic batch-size (DeepStream >= 6.1).

    CLI Flags for export_yolox.py

    • --simplify: Simplifies the ONNX model (recommended for DeepStream >= 6.0).
    • --dynamic: Enables dynamic batch-size (DeepStream >= 6.1).
    • --batch <N>: Sets a static batch-size (e.g., --batch 4).
    • --opset <N>: Sets the ONNX opset. For DeepStream 5.1, remove --dynamic and use --opset 12 or lower (default is 11).

  7. Configure and run INT8 calibration

    master

    To run the INT8 calibration process, set the environment variables for the image path and batch size, update your config_infer file to use the INT8 engine and calibration table, and then run the application.

    Note on Accuracy: NVIDIA recommends at least 500 images. Using 1000 images is recommended for better accuracy. Increasing INT8_CALIB_BATCH_SIZE can improve accuracy and calibration speed, but should be tuned according to your GPU memory.

  8. Install TensorRT and cuDNN for DeepStream 6.2

    master

    Install the specific TensorRT and cuDNN versions required for DeepStream 6.2 (CUDA 11.8).

    1. Add the NVIDIA repository:
      sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/3bf863cc.pub
sudo add-apt-repository "deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/ /"
sudo apt-get update
    2. Install the packages:
      sudo apt-get install libnvinfer8=8.5.2-1+cuda11.8 libnvinfer-plugin8=8.5.2-1+cuda11.8 libnvparsers8=8.5.2-1+cuda11.8 libnvonnxparsers8=8.5.2-1+cuda11.8 libnvinfer-bin=8.5.2-1+cuda11.8 libnvinfer-dev=8.5.2-1+cuda11.8 libnvinfer-plugin-dev=8.5.2-1+cuda11.8 libnvparsers-dev=8.5.2-1+cuda11.8 libnvonnxparsers-dev=8.5.2-1+cuda11.8 libnvinfer-samples=8.5.2-1+cuda11.8 libcudnn8=8.7.0.84-1+cuda11.8 libcudnn8-dev=8.7.0.84-1+cuda11.8 python3-libnvinfer=8.5.2-1+cuda11.8 python3-libnvinfer-dev=8.5.2-1+cuda11.8
    3. Hold the packages to prevent accidental updates:
      sudo apt-mark hold libnvinfer* libnvparsers* libnvonnxparsers* libcudnn8* python3-libnvinfer*
    sudo apt-get install libnvinfer8=8.5.2-1+cuda11.8 libnvinfer-plugin8=8.5.2-1+cuda11.8 libnvparsers8=8.5.2-1+cuda11.8 libnvonnxparsers8=8.5.2-1+cuda11.8 libnvinfer-bin=8.5.2-1+cuda11.8 libnvinfer-dev=8.5.2-1+cuda11.8 libnvinfer-plugin-dev=8.5.2-1+cuda11.8 libnvparsers-dev=8.5.2-1+cuda11.8 libnvonnxparsers-dev=8.5.2-1+cuda11.8 libnvinfer-samples=8.5.2-1+cuda11.8 libcudnn8=8.7.0.84-1+cuda11.8 libcudnn8-dev=8.7.0.84-1+cuda11.8 python3-libnvinfer=8.5.2-1+cuda11.8 python3-libnvinfer-dev=8.5.2-1+cuda11.8

  9. Convert YOLOv9 models to ONNX

    master

    To use YOLOv9 with DeepStream, you must convert the PyTorch (.pt) model to ONNX format using the provided export_yoloV9.py script.

    Prerequisites:

    1. Clone the official YOLOv9 repository and install its requirements: pip3 install -r requirements.txt and pip3 install onnx onnxslim onnxruntime.
    2. Copy export_yoloV9.py from the DeepStream-Yolo/utils directory into your YOLOv9 working folder.
    3. Download your desired .pt model (e.g., yolov9-s-converted.pt).

    Conversion Command: Run the export script with the appropriate flags for your DeepStream version and requirements.

  11. Install TensorRT for DeepStream 7.0

    master

    Install the specific TensorRT version required for DeepStream 7.0 by adding the NVIDIA CUDA repository and installing the libnvinfer* and related packages. It is critical to use apt-mark hold on the installed packages to prevent accidental updates that could break the DeepStream environment.

    sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/3bf863cc.pub
sudo add-apt-repository "deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/ /"
sudo apt-get update
sudo apt-get install --no-install-recommends libnvinfer-lean8=8.6.1.6-1+cuda12.0 libnvinfer-vc-plugin8=8.6.1.6-1+cuda12.0 libnvinfer-headers-dev=8.6.1.6-1+cuda12.0 libnvinfer-dev=8.6.1.6-1+cuda12.0 libnvinfer-headers-plugin-dev=8.6.1.6-1+cuda12.0 libnvinfer-plugin-dev=8.6.1.6-1+cuda12.0 libnvonnxparsers-dev=8.6.1.6-1+cuda12.0 libnvinfer-lean-dev=8.6.1.6-1+cuda12.0 libnvparsers-dev=8.6.1.6-1+cuda12.0 python3-libnvinfer-lean=8.6.1.6-1+cuda12.0 python3-libnvinfer-dispatch=8.6.1.6-1+cuda12.0 uff-converter-tf=8.6.1.6-1+cuda12.0 onnx-graphsurgeon=8.6.1.6-1+cuda12.0 libnvinfer-bin=8.6.1.6-1+cuda12.0 libnvinfer-dispatch-dev=8.6.1.6-1+cuda12.0 libnvinfer-dispatch8=8.6.1.6-1+cuda12.0 libnvonnxparsers-dev=8.6.1.6-1+cuda12.0 libnvonnxparsers8=8.6.1.6-1+cuda12.0 libnvinfer-vc-plugin-dev=8.6.1.6-1+cuda12.0 libnvinfer-samples=8.6.1.6-1+cuda12.0
sudo apt-mark hold libnvinfer* libnvparsers* libnvonnxparsers* libcudnn8* python3-libnvinfer* uff-converter-tf* onnx-graphsurgeon*

  12. Compile the DeepStream-Yolo library

    master

    Before running the application, you must compile the custom inference implementation. You must set the CUDA_VER environment variable to match the CUDA version bundled with your DeepStream installation.

    CUDA Version Mapping (x86):

    • DeepStream 8.0: 12.8
    • DeepStream 7.1: 12.6
    • DeepStream 7.0 / 6.4: 12.2
    • DeepStream 6.3: 12.1
    • DeepStream 6.2: 11.8
    • DeepStream 6.1.1: 11.7
    • DeepStream 6.1: 11.6
    • DeepStream 6.0.1 / 6.0: 11.4
    • DeepStream 5.1: 11.1

    CUDA Version Mapping (Jetson):

    • DeepStream 8.0: 13.0
    • DeepStream 7.1: 12.6
    • DeepStream 7.0 / 6.4: 12.2
    • DeepStream 6.3 / 6.2 / 6.1.1 / 6.1: 11.4
    • DeepStream 6.0.1 / 6.0 / 5.1: 10.2
    export CUDA_VER=XY.Z
make -C nvdsinfer_custom_impl_Yolo clean && make -C nvdsinfer_custom_impl_Yolo