DL Streamer Documentation

The Pose Estimation Compose sample is a multi-stream pipeline that runs four different YOLO pose models (YOLO26n, YOLO11n, YOLOv8n, and YOLOv8l) in parallel. Each model processes the same input video, and the pipeline composites the annotated outputs into a single 2×2 mosaic video.

Pipeline Workflow:

1. Decodes the input video four times.
2. Detects people and estimates pose keypoints using gvadetect with different YOLO pose models per stream.
3. Annotates each stream with keypoints and custom model-name text using gvawatermark.
4. Scales each stream to a tile size via GPU using vapostproc.
5. Composites the four streams into a 2×2 grid using vacompositor.
6. Encodes the mosaic to H.264 and saves it as an MP4 using vah264enc → mp4mux → filesink.

Key DL Streamer Elements Used:

• filesrc: Reads input video.
• decodebin3: Decodes video (hardware-accelerated).
• gvadetect: Performs inference for YOLO pose estimation.
• gvawatermark: Overlays keypoints and text.
• vapostproc: GPU-accelerated scaling.
• vacompositor: GPU-accelerated 2×2 grid compositing.
• vah264enc: Hardware H.264 encoding.
• mp4mux: MP4 container muxing.
• filesink: Writes output to disk.

This sample demonstrates a vision AI processing pipeline that detects and tracks people in a video stream. It uses a multi-stage pipeline to decode video, perform object detection, extract appearance features, and maintain identities across frames.

Pipeline Workflow:

1. Decode: Reads video from a local file using filesrc and decodebin3.
2. Detect: Uses gvadetect with the YOLO26m model to find people.
3. Extract Features: Uses gvainference with the Mars-Small-128 model to extract 128-dimensional appearance features for each detected person.
4. Track: Uses gvatrack with the Deep SORT algorithm (Kalman filter + cosine distance matching) to track individuals across frames.
5. Annotate: Uses gvawatermark to overlay bounding boxes and tracking IDs on the video.
6. Metadata: Uses gvametaconvert and gvametapublish to export detection data as JSON Lines.
7. Encode: Uses vah264enc, mp4mux, and filesink to save the annotated video as an MP4 file.

DLStreamer provides a variety of Python samples designed to demonstrate how to integrate GStreamer pipelines with custom Python logic. These samples cover common use cases such as integrating Hugging Face models, handling ONVIF camera discovery, building dynamic pipelines, and using Vision Language Models (VLM) for GenAI-driven alerts and classification.

Key capabilities demonstrated in these samples include:

• Model Integration: Using PyTorch models from Hugging Face and other hubs within DLStreamer inference elements.
• Custom Logic: Adding Python callbacks to process detection results or creating custom Python elements to filter frames.
• Advanced Analytics: Implementing tripwire counting, line hogging detection, and open-vocabulary classification.
• Metadata & Visualization: Using the gvawatermark element and the watermark metadata API to draw polygons, lines, circles, and text on video frames.
• GenAI & VLM: Triggering alerts based on VLM identification and combining standard CV object detection with VLM-enhanced classification.

The following sample applications demonstrate various DL Streamer capabilities, ranging from simple object detection to complex multi-stream composition and VLM-based safety monitoring.

Note: Each sample directory contains its own README with specific setup instructions, model export scripts, and usage examples. You should navigate to the specific sample folder for detailed implementation details.

| Sample | Description |
|--------|-------------|
| [deepstream_python_conversion](deepstream_python_conversion/) | DL Streamer equivalent of NVIDIA's deepstream-test4 — detects objects with YOLO11n, overlays bounding boxes, and publishes detection metadata to file/Kafka/MQTT. |
| [deepstream_cpp_conversion](deepstream_cpp_conversion/) | C++ conversion of NVIDIA's DeepStream LPR app — detects license plates with YOLOv11, tracks them, and recognizes plate text with PaddleOCR. Outputs annotated video and JSONL metadata. |
| [license_plate_recognition](license_plate_recognition/) | Detects license plates with YOLOv11 and recognizes plate text with PaddleOCR. Outputs JSON Lines results and annotated video. |
| [multi_stream_compose](multi_stream_compose/) | Processes 4 video streams with shared YOLO11s detection, composites into a 2×2 GPU-accelerated mosaic streamed via WebRTC, with on-demand per-stream recording. |
| [people_detection_tracking](people_detection_tracking/) | Detects and tracks people using YOLO26m + Deep SORT with Mars-Small-128 re-ID model. Outputs annotated video with tracking IDs and JSON metadata. |
| [pose_estimation_compose](pose_estimation_compose/) | Runs 4 YOLO pose models in parallel on the same video, annotates each with keypoints and model name, and composites into a 2×2 mosaic. |
| [safety_compliance](safety_compliance/) | Monitors construction site safety — detects workers with YOLO26m, tracks them, uses Qwen2.5-VL to verify helmet and harness compliance, and generates alerts for violations. |
| [smart_nvr](smart_nvr/) | Event-based smart recording — detects people with YOLO11n and records video only when a person is present, producing sequentially numbered MP4 files. |

DL Streamer provides a wide variety of samples categorized by implementation method and use case:

1. gst_launch command-line samples

These samples use the gst-launch-1.0 utility to construct GStreamer pipelines via the CLI. They cover a broad range of tasks including:

• Audio: Event detection (gvaaudiodetect), transcription (gvaaudiotranscribe), and metadata publishing.
• Vision/Detection: YOLO object detection, face detection/classification, instance segmentation, and depth estimation.
• 3D/LiDAR/Radar: LiDAR parsing (g3dlidarparse), 3D object fusion (g3dobjectfuser), and radar signal processing (g3dradarprocess).
• Advanced AI: Vision-Language Models (VLM) for video summarization (gvagenai), and Geti™ model deployment.
• Pipeline Control: FPS throttling (gvafpsthrottle), ROI definition (gvaattachroi), and multi-stream handling (gvastreammux/gvastreamdemux).

2. C++ samples

Demonstrate how to construct pipelines and use C++ callbacks to access frame metadata and visualize results (e.g., draw_face_attributes).

3. Python samples

Demonstrate high-level integration using Python, including:

• Model Management: Downloading and exporting models from Hugging Face to OpenVINO IR.
• Custom Logic: Using gvaanalytics for tripwires, managing hardware (Open/Close valve) via callbacks, and building Smart NVRs.
• VLM & Open Vocabulary: Prompt-based object detection and VLM-assisted self-checkout.
• Metadata: Using the watermark metadata API to draw primitives on frames.

4. Performance & Benchmarking

• Benchmarks: Measuring throughput for single or multi-channel pipelines.
• E2E Performance: Comparing DL Streamer throughput against OpenCV + OpenVINO on Intel Core Ultra processors.

5. Specialized Samples

• DeepStream Coexistence: Running pipelines on both DL Streamer and DeepStream.
• Auto-Generated Samples: Conversions from NVIDIA DeepStream (e.g., deepstream-test4 equivalent) and optimized workflows for Intel Core Ultra (e.g., License Plate Recognition).

The gvawatermark element overlays metadata (such as bounding boxes, labels, and keypoints) onto video frames to visualize inference results. It supports various video formats including BGRx, BGRA, BGR, NV12, and I420 across different memory types like DMABuf, VASurface, and VAMemory.

Key Capabilities

• Visual Overlays: Displays bounding boxes, text labels, and keypoints.
• ROI Filtering: Show or hide specific object types (e.g., only show 'person').
• Privacy Protection: Includes a blurring feature to obscure specific Regions of Interest (ROIs).
• Custom Text: Allows displaying custom full-frame text at specific coordinates.
• FPS Display: Can display average FPS if a gvafpscounter element is present in the pipeline.

# Example usage of gvawatermark with custom configuration
# (Note: This is a conceptual representation of how the element is used in a GStreamer pipeline)
... ! gvawatermark displ-cfg="show-labels=true,font-scale=0.8,color-idx=1" ! ...