Savant Framework

Savant is an open-source, high-level framework designed for building real-time, streaming, and highly efficient multimedia AI applications on the Nvidia stack. It provides an abstraction layer over Nvidia DeepStream, allowing developers to build dynamic, fault-tolerant inference pipelines for both data centers and edge devices without low-level programming.

Key features include:

• High performance on Nvidia hardware (dGPU and Jetson).
• Declarative pipeline construction using YAML blocks and Python functions.
• Support for heterogeneous pipelines with diverse models and data sources.
• Observability via OpenTelemetry and Prometheus.
• Support for hybrid edge/datacenter applications with a single codebase.

Savant is an open-source, high-level framework written in Python, C++, and Rust designed for building real-time, streaming, and highly efficient computer vision AI applications. It is built on top of the Nvidia DeepStream SDK to provide a high-level abstraction that hides complexity while leveraging Nvidia's best technologies for both data center GPUs and Jetson edge devices.

Key characteristics:

• Target Hardware: Supports discrete Nvidia GPUs (Data Center) and Nvidia Jetson (Edge).
• Pipeline Model: Pipelines are defined as a sequence of declarative blocks using YAML configuration, combined with user-defined logic implemented in Python.
• Core Purpose: Enables rapid development of dynamic, fault-tolerant inference pipelines.

The Savant Client SDK is a Python library designed to simplify the development of sources and sinks using pure Python. It allows developers to ingest frames and metadata into a running Savant module and receive processed results.

Key use cases include:

• Developing integration tests for Pipelines (QA).
• Implementing custom source adapters without needing deep knowledge of streaming technology.
• Implementing custom sink adapters without needing deep knowledge of streaming technology.
• Enabling remote development.

The Facial ReID sample demonstrates a pipeline for facial re-identification using YOLOV8-Face (for detection and landmarks) and Adaface (for recognition). It is designed for use cases like doorbell security systems.

The sample is divided into two functional modules:

1. Index Builder: Processes a gallery of images to detect faces, perform preprocessing, and run recognition inference. It stores the resulting feature vectors in an hnswlib index and saves cropped face images to disk.
2. Demo: Loads the generated index and cropped images, then processes a live video stream to detect and match faces, displaying matches in a side panel.

Note: The first run takes a long time (30-40 minutes) because the models must be compiled into TensorRT engines.

Savant uses DeepStream NvInfer to run inference via TensorRT-optimized neural networks. There are four primary types of inference units available, each serving a different purpose in a computer vision pipeline:

• Detector Unit (nvinfer@detector): Used for models that produce bounding boxes, classes, and confidence scores (e.g., object detection).
• Attribute Model Unit (nvinfer@attribute_model): Used for models that produce specific attributes from an input (e.g., gender, age, or facial landmarks). Note that Classifier Unit (nvinfer@classifier_model) is an alias for this unit.
• Complex Model Unit (nvinfer@complex_model): Used for multi-task models that produce both bounding boxes/classes and attributes (e.g., keypoints) simultaneously.

When reviewing Savant samples or documentation, platform support is indicated using the following labels:

• X86 + L4T: Supported on both x86 and Jetson (L4T) platforms.
• X86 only: Supported only on x86 platforms.
• L4T only: Supported only on Jetson (L4T) platforms.

If no platform is specified, the sample is assumed to be implemented for the X86 platform.

All inter-service communication in the Meta Merge sample uses DEALER/ROUTER sockets over IPC. The connection pattern (whether a service binds or connects) is critical for the pipeline to function correctly.