DOCSEARCHbyAlgolia
Submit Docs

DeepStream Services Library (DSL)

repository·master·Indexed 43 days ago

https://github.com/prominenceai/deepstream-services-library

A high-level, on-demand service layer for NVIDIA DeepStream and GStreamer, enabling the creation of complex Intelligent Audio and Video Analytics (IAVA) pipelines. Written in C++ 17 with an extern "C" API, it provides support for both C/C++ and Python applications. The library offers abstractions for managing pipeline components, branches, custom GStreamer element wrapping, and internal queue management to decouple processing threads.

Tokens
203.8K
Snippets
619
Records
844
Agent score
56%

What's inside DeepStream Services Library

  1. Overview of Intelligent Audio and Video Analytics (IAVA)

    master

    The DeepStream Services Library (DSL) is a collection of on-demand DeepStream Pipeline services designed for Intelligent Audio and Video Analytics (IAVA).

    Key characteristics:

    • Language Support: Written in C++ 17 with an extern "C" API, making it accessible from both C/C++ and Python applications.
    • Foundation: Built on the NVIDIA® DeepStream SDK and the open-source GStreamer framework.
    • Purpose: Provides high-level abstractions for creating complex streaming analytics pipelines for AI-based video and image understanding.

  2. Overview of Azure MQTT Protocol Adapter Libraries

    master

    NVIDIA provides two Azure MQTT protocol libraries located in /opt/nvidia/deepstream/deepstream/lib for use with the DSL Message Sink and Message Broker components:

    1. libnvds_azure_proto.so (Device Client Protocol): Used for sending messages from the device to an Azure IoT Hub instance. This requires minimal setup.
    2. libnvds_azure_edge_proto.so (Module Client Protocol): Used for bidirectional device-server messaging. Applications using this library must be run within a deployed Docker container.

  3. Overview of Object Detection Event (ODE) Services

    master

    The Object Detection Event (ODE) services provide a suite of tools for analyzing and reacting to detected objects in a video stream. These services include:

    • ODE Pad Probe Handlers: Intercepts data at specific GStreamer pads.
    • ODE Triggers: Evaluates specific criteria (e.g., class ID, distance, persistence, intersection) to fire events.
    • ODE Actions: Executes a task when a trigger is met (e.g., printing data, displaying metadata, capturing frames, or sending emails).
    • ODE Areas: Defines spatial regions (polygons) for inclusion or exclusion.
    • ODE Accumulators: Tracks and aggregates occurrences over time.
    • ODE Heat Mappers: Visualizes object density/occurrence using color palettes.

  5. Component API Reference Overview

    master

    The Component API provides the foundational interface for managing elements within the DeepStream services library. It serves as a high-level abstraction over GStreamer (GST) elements, allowing for the creation of custom components, management of element lifecycles, and integration into the broader Pipeline.

    For specific details on individual service components, refer to the specialized API documentation for:

    • Pipeline
    • Player
    • Source
    • Tap
    • Dewarper
    • Preprocessor
    • Inference Engine and Server
    • Tracker
    • Segmentation Visualizer
    • Tiler
    • Demuxer and Splitter Tees
    • Remuxer
    • On-Screen Display
    • Sink
    • Branch
    • GST Element
    • Pad Probe Handler
    • ODE (Trigger, Accumulator, Action, Area, Heat-Mapper)
    • Mailer
    • WebSocket Server
    • Message Broker
    • Info API

  7. What is the Remuxer and how does it work?

    master

    The Remuxer is an aggregate component used to implement Selective Parallel Inference. It links several sub-components together: a Demuxer, Tees, Streammuxers, Inference Branches, and a Metamuxer.

    Core Workflow

    1. Demuxing: Incoming streams are split.
    2. Inference Branches: Streams are routed to specific inference branches. An Inference branch can be a single Primary Inference Component or a Branch Component containing multiple inference components.
    3. Metamuxing: The Metamuxer recombines the processed streams. The Remuxer automatically manages the Metamuxer configuration file (stored in /tmp/) based on which streams are assigned to which inference branches.

  8. What is the Dewarper component and how does it work?

    master

    The Dewarper component is used to dewarp a Source component's output-buffer stream. It is built on NVIDIA's Gst-nvdewarper plugin and supports three projection types:

    1. PushBroom: Used for dewarping 360° camera input.
    2. VertRadCyl: Used for dewarping 360° camera input.
    3. PerspectivePerspective: All dewarping parameters must be defined in the configuration file (the camera_id parameter is ignored for this type).

    Dewarpers are derived from the component class, meaning they inherit all standard component methods.

    Relationship with Sources: The relationship between Sources and Dewarpers is one-to-one. Once a Dewarper is added to a Source, it must be removed before it can be used with a different Source.

  11. Use Multi-Source Tiler to grid video streams

    master

    When using multiple sources, the Stream Muxer batches them into a single stream. To view these multiple sources simultaneously, you must use a Tiler component to transform the batched stream into a 2D grid array of tiles (one per source) before sending it to an On-Screen Display (OSD) or Sink.

    Key Features:

    • Dimensions: Configure the Tiler with specific width and height in pixels, as well as rows and columns settings.
    • Source Visibility: Use dsl_tiler_source_show_set to show a single specific source, and dsl_tiler_source_show_all to return to the full tiled view.
    • Metadata Processing: You can add Pad Probe Handlers to the Tiler's sink-pad (input) or source-pad (output) to process metadata.

    Note: Tilers output a single stream, so the OSD and Sinks should be added directly to the Pipeline or Branch operating on the Tiler's output.

    # assumes all components have been created first
retval = dsl_pipeline_component_add_many('my-pipeline', 
    ['src-1', 'src-2', 'pgie', 'tiler', 'osd', 'rtsp-sink', 'window-sink', None])

  12. Implement Smart Recording with Record Tap and Record Sink

    master

    Smart recording provides cached-video recording by buffering the last N seconds of video in a fixed-size cache. There are two primary components for this:

    1. Record Tap: Taps into the pre-decoded stream to record the original video. This is added directly to an RTSP Source component.
    2. Record Sink: Encodes the decoded, inferred-on, and optionally annotated stream. This is added to a Pipeline or Branch.

    Recording sessions can be started using a start point (seconds before current time) and a duration (seconds from current time). The total duration is start + duration seconds.

    Applications can trigger recordings based on:

    • Object Detection Events (ODE): Using ODE Triggers and Start-Record Actions.
    • User/Application Demand: Manual invocation via API.

    A client callback can be attached to either component to notify the application when a recording starts (DSL_RECORDING_EVENT_START) or completes (DSL_RECORDING_EVENT_END).