DOCSEARCHbyAlgolia
Submit Docs

NVIDIA DeepStream Reference Applications

repository·master·Indexed 23 days ago

https://github.com/nvidia-ai-iot/deepstream_reference_apps

A collection of reference applications for video analytics using TensorRT and DeepStream SDK. Includes implementations for anomaly detection with the dsdirection plugin, parallel multi-model inference using Triton, 3D body pose estimation, and multi-modal sensor fusion (LiDAR, RADAR, and Camera) via the DS3D framework.

Tokens
33.8K
Snippets
80
Records
136
Agent score
79%

What's inside deepstream_reference_apps

  1. Overview of DeepStream 9.0 Reference Applications

    master

    This repository provides reference applications for video analytics tasks using TensorRT and DeepStream SDK 9.0. Available samples include:

    • Anomaly Detection: Uses an auxiliary dsdirection plugin for anomaly detection capabilities.
    • Runtime Source Addition Deletion: Demonstrates managing video sources in a live pipeline.
    • MaskTracker: Demonstrates multi-object tracking and segmentation using SAM2.
    • Single-View 3D Tracking: Reconstructs 3D human models in world coordinates under occlusion.
    • Multi-View 3D Tracking: Designed for large-scale, calibrated camera networks.
    • Parallel Multiple Models Inferencing: Implements multiple models inferencing in parallel using DeepStream APIs.
    • Bodypose 3D Model Inferencing: Demonstrates customizing multiple input layer model preprocessing and 3D bodypose model postprocessing.
    • Video Buffers sharing through IPC: Demonstrates sharing video buffers over IPC and changing output video buffers.
    • Multiple Dynamic Sources with Single Decoder: Demonstrates usage of nvdsdynamicsrcbin to adapt to high decoder throughput scenarios.
    • Custom Video Tiling Config: Demonstrates using custom-tile-config of nvmultistreamtiler to customize tiling positions and sizes.
    • DeepStream VLLM Plugin: A GStreamer plugin integrating Vision-Language Models (VLM) via VLLM for real-time video understanding.

  2. What is DS3D Multi-modal sensor fusion

    master

    The deepstream-3d-lidar-sensor-fusion application uses the DS3D framework to implement multi-modal sensor fusion pipelines (LiDAR, RADAR, and Camera).

    Key components of the DS3D framework used in this application include:

    • Camera processing: Uses DeepStream's generic 2D video pipeline with batchMeta.
    • ds3d::dataloader: Handles LiDAR capture with pre-processing.
    • ds3d::databridge: Converts DeepStream NvBufSurface and GstNvDsPreProcessBatchMeta into shaped tensor data (s3d::Frame2DGuard and ds3d::FrameGuard formats) and embeds key-value pairs in ds3d::datamap.
    • ds3d::mixer: Merges camera, LiDAR, and other sensor data into ds3d::datamap.
    • ds3d::datatfiler + libnvds_tritoninferfilter.so: Performs multi-modal inference on the ds3d::datamap using Triton.
    • ds3d::datasink + ds3d_gles_ensemble_render: Visualizes 3D detection results via a multi-view display.

  4. How parallel inferencing branches work

    master

    The application constructs a pipeline where multiple models run in parallel. To implement this:

    1. Unique IDs: Every GIE (Inference Engine) must have a unique unique-id. The gst-nvdsmetamux module uses these IDs to identify which metadata belongs to which model.
    2. Branching: The application creates new inferencing branches for designated primary GIEs.
    3. Secondary GIEs: Secondary GIEs must identify the primary GIE they operate on by setting the operate-on-gie-id property in their respective nvinfer or nvinferserver configuration files.
    4. Source Selection: Use the branch configuration group to route specific source IDs to specific inferencing branches.

  5. How DeepStream Parallel Inference works

    master

    DeepStream Parallel Inference is an application designed to run multiple models in parallel. The pipeline architecture uses the following components:

    • tee: Sends batched buffers to different models simultaneously.
    • nvdsmetamux: Muxes the batch metadata from the different parallel models back together.

    Configuration is handled via YAML files:

    • Application Configuration (.yml): Defines the overall pipeline structure.
    • Preprocess Configuration: Allows the user to select specific source ids that require inference.
    • nvdsmetamux Configuration: Allows the user to select specific source ids that need to be muxed.

  6. How Single-View 3D Tracking works in DeepStream

    master

    Single-View 3D Tracking (SV3DT) estimates and tracks object states in a 3D physical world using a static camera's camera matrix and human model (configured in configs/camInfo.yml).

    Key capabilities:

    • Recovers complete bounding boxes, foot locations, and body convex hulls even during partial occlusions.
    • Uses a 2D pose estimator to find human key-points on the image plane.
    • Uses height and waist key-points as anchors to estimate 3D human height.

    Configuration Options: In configs/config_tracker_NvDCF_accuracy_3D.yml, you can set poseEstimatorType: 0 to disable the pose estimator. In this mode, the algorithm uses 2D detection bounding boxes and a human model with a fixed height, estimating 3D location by matching the head with the top edge of the 2D bounding box.

  7. DeepStream Retail Analytics Pipeline Architecture

    master

    The application performs real-time Intelligent Video Analytics (IVA) to detect customers and identify if they are carrying shopping baskets.

    Pipeline Components:

    • Primary Detector: PeopleNet Pre-Trained Model (PTM) from NGC.
    • Secondary Detector: Custom classification model (trained via TAO) to classify people with/without baskets.
    • Object Tracker: NvDCF tracker.
    • Message Converter: Custom nvmsgconv library to generate custom payloads from inference data.
    • Message Broker: Relays inference data to a Kafka server.
    • Data Flow: Inference output $\rightarrow$ Kafka $\rightarrow$ Time-Series Database (TSDB) $\rightarrow$ Django REST API $\rightarrow$ Front-end Dashboard.

  8. How the MV3DT Auto-Configurator works

    master

    The auto-configurator automates the creation of a complete MV3DT configuration set by performing the following tasks:

    1. Dataset Detection: Scans videos/ and camInfo/ directories to determine the number of cameras and locate calibration files.
    2. Pipeline Generation: Uses config_templates/config_deepstream.txt as a template to generate a final $EXPERIMENT_DIR/config_deepstream.txt with correct source URIs, batch sizes, and display grid layouts.
    3. Tracker Extension: Injects MV3DT-specific sections into your 2D tracker configuration, specifically:
      • ObjectModelProjection: For 3D model projection.
      • MultiViewAssociator: For multi-view association.
      • Communicator: For inter-camera communication.
    4. Inter-camera Communication: Generates $EXPERIMENT_DIR/pub_sub_info_config_0.yml to define how cameras publish and subscribe to multi-view data.
    5. Overrides (Optional): Applies override files to customize tracker settings for specific datasets or requirements.

  9. Understand the Runtime Source Addition/Deletion pipeline logic

    master

    This application demonstrates a dynamic pipeline where sources are added and removed during execution.

    Pipeline Architecture

    For a single source, the pipeline follows this flow: uridecodebin -> nvstreammux -> nvinfer -> nvtracker -> nvtiler -> nvvideoconvert -> nvdsosd -> displaysink

    Dynamic Behavior

    • Addition: After a timeout, a new source is added periodically. All pipeline components are reconfigured to accommodate the new source.
    • Deletion: Once the pipeline reaches MAX_NUM_SOURCES, sources are deleted periodically until only a single source remains.
    • Termination: The application exits when the final source reaches End of Stream (EOS) or if the last remaining source is deleted.

  10. Customize tiling positions and sizes with nvmultistreamtiler

    master
    The deepstream-custom-tile-config sample demonstrates how to use the custom-tile-config property of the nvmultistreamtiler element. This allows you to manually define the tiling positions and sizes for multiple video streams within the display window using a CustomTileConfig struct, rather than relying on default tiling layouts.

  11. How back-to-back detectors work in DeepStream

    master

    The Back-to-Back Detectors application demonstrates how to chain multiple inference elements (nvinfer or nvinferserver) serially in a DeepStream pipeline. The pipeline structure places two inference instances after nvstreammux and before the display components.

    Detector Modes

    1. Primary Detector: The first instance (e.g., Person/Vehicle/Bicycle/RoadSign) always acts as the primary detector.
    2. Secondary Detector: The second instance (e.g., Face Detection) can operate in two modes:
      • Secondary Mode (Default): Operates only on objects detected by the primary detector (e.g., detecting faces within a detected person's bounding box).
      • Primary Mode: Operates on the full frame. To switch the second detector to primary mode, you must change the macro SECOND_DETECTOR_IS_SECONDARY to 0 in the source code.

    This pattern allows for hierarchical inference, such as detecting a vehicle and then detecting a license plate within that vehicle.

  12. Deploy TAO classification models in DeepStream

    master

    Once you have exported your model, there are two primary ways to deploy it within a DeepStream pipeline:

    1. Direct Usage: Use the exported .etlt model file directly with DeepStream.
    2. Engine Usage (Recommended): Use tao-converter to generate a device-specific engine file. You must then point your DeepStream configuration file to this generated engine file for the inference engine component. See basket_classifier.yml in the configs directory for a reference implementation.