ZED SDK Documentation

The ZED SDK provides capabilities to create video streams from a ZED camera that can be read by remote applications for viewing or processing. The repository includes three primary sample implementations:

• ZED Stream Single Sender: Establishes a network connection with a single camera, encodes the live video stream, and transmits it to a remote client. This is suitable for remote video capture, computer vision, or teleoperation.
• ZED Stream Multiple Sender: Enables multiple cameras to run in their own threads, each encoding and transmitting its own live video stream to a remote client.
• ZED Stream Receiver: Provides the logic to receive and decode video data sent from a remote ZED stream.

The PyTorch YOLO Detector sample allows you to detect custom objects using official PyTorch implementations of YOLO (including YOLOv8, v9, v10, v11, v12, and v26) with a ZED camera. The module integrates detected 2D objects into the ZED SDK to provide 3D information and real-time tracking.

Key Features:

• 3D OpenGL point cloud visualization.
• 3D bounding boxes around detected objects.
• Configurable object classes and confidence thresholds.
• Real-time tracking of detected objects.

The Live Global Localization sample demonstrates how to achieve global-scale localization by fusing visual odometry from the ZED SDK with external GNSS data in real-time. This is designed for applications like autonomous robotics and drone navigation.

Key Features:

• 3D visualization of the camera's path in an OpenGL window.
• Real-time display of path data (translation and rotation).
• Fused path visualization on a web-based map.
• Exporting of KML files containing both the fused trajectory and raw GNSS data.

The CUDA Refocus sample demonstrates how to grab and process images and depth data using a custom CUDA kernel for depth-based focusing.

It implements a layered depth-of-field rendering effect using a Gaussian blur where the kernel size is dynamically determined by the actual depth of the pixels. The implementation is based on separable convolution CUDA samples to ensure optimal performance.

The ZED SDK provides a Positional Tracking module that enables accurate and reliable tracking of a ZED stereo camera's position and orientation in 3D space. This module is used to determine the device's pose (location and rotation) relative to its environment.

Key capabilities demonstrated in the repository include:

• Positional Tracking: Leveraging the stereo camera to track device movement in 3D space.
• FBX Data Export: Exporting the generated tracking data into the FBX format for use in external 3D software or animation pipelines.

The ZED SDK is a cross-platform library for Stereolabs ZED cameras, providing a spatial perception platform with real-time performance. Key capabilities include:

• Depth Sensing: Capturing 3D point clouds and depth data.
• Object Detection: Identifying objects within the camera's field of view.
• Body Tracking: Tracking human movement and poses.
• Positional Tracking: Real-time tracking of the camera's position and orientation.
• Global Localization: Fusing positional tracking with GNSS data for global positioning.
• Spatial Mapping: Capturing 3D meshes and point clouds.
• Camera Control: Adjusting camera parameters.
• Plane Detection: Identifying flat surfaces in a scene.
• Multi Camera Fusion: Combining data from multiple ZED cameras.

The Global Localization Data Recording sample allows you to record data necessary for global localization on real-world maps using a ZED camera. It is designed for developers working in autonomous driving, robotics, and drone navigation.

Outputs generated during recording:

• SVO file: Contains the camera data.
• JSON file: Contains the pre-recorded GNSS data (used for playback in the playback sample).
• KML files: Used to display raw GNSS data and fused positions on real-world maps like Google Maps after capture.

Key Features:

• 3D OpenGL window displaying the camera's path.
• Real-time display of path data (translation and rotation).
• Generation of KML, SVO, and JSON files for post-capture analysis and map server integration.

The ZED One samples demonstrate various capabilities of the ZED One mono camera using the ZED SDK. Key functionalities include:

• Live Video: Capturing and displaying live video with camera settings control.
• SVO Recording: Recording video to the SVO format for later playback.
• SVO Playback: Playing back recorded SVO files and controlling playback.
• SVO External Data: Embedding and retrieving custom data within SVO files.
• Streaming: Sending live video over a network (Sender) and receiving/displaying remote streams (Receiver).
• Custom Inference: Retrieving tensors directly from the ZED SDK to run AI models.

The ZED SDK Body Tracking module allows for the detection and tracking of human bodies in 3D space. It provides real-time skeletal data and spatial tracking capabilities.

Key Features:

• Skeletal Visualization: Renders human body skeletons overlaid on camera images.
• Temporal & Spatial Tracking: Tracks individual bodies continuously across both time and 3D space.
• Bounding Box Toggle: Users can toggle the display of body bounding boxes by pressing the b key during runtime.