DOCSEARCHbyAlgolia
Submit Docs

Orbbec SDK

repository·main·Indexed 43 days ago

https://github.com/orbbec/orbbecsdk

A unified interface for interacting with Orbbec range sensors and cameras, supporting modern UVC-compliant products and legacy OpenNI protocol devices. The SDK provides a C High Level API for managing device lifecycles, configuring stream profiles (Depth, Color, and IR), performing stream alignment via Align Filter, monitoring device connection changes, and managing firmware upgrades.

Tokens
95K
Snippets
214
Records
266
Agent score
51%

What's inside Orbbec SDK

  1. Overview of Orbbec SDK features and capabilities

    main

    The Orbbec SDK is a cross-platform (Windows, Android, Linux) software development kit designed for 3D sensing cameras, including structured light, binocular, and iToF devices. It provides both high-level APIs for rapid development and low-level APIs for granular control.

    Core Capabilities:

    • Camera Access: Configuration and access for Depth, RGB, and Sensor (Gyroscope/Accelerometer) data.
    • Data Processing: Frame synchronization, stream alignment, and point cloud generation.
    • Algorithmic Support: Built-in filtering and algorithmic capabilities.
    • Cross-Platform: Support for multiple operating systems and wrappers.

  2. Overview of Orbbec SDK core capabilities

    main

    The Orbbec SDK is a cross-platform (Windows, Android, Linux) software development kit designed for Orbbec 3D sensing cameras (Structured Light, Binocular, iToF). It provides both high-level APIs for rapid development and low-level APIs for maximum flexibility and performance.

    Core Capabilities:

    • Access and parameter configuration for Depth, RGB, and Sensor (Gyroscope/Accelerometer) components.
    • Control over frame synchronization and alignment.
    • Point cloud data generation.
    • Algorithmic capabilities such as filtering.
    • Support for multiple operating systems and wrappers.

  3. Understand the OrbbecViewer Interface

    main

    The OrbbecViewer interface is divided into three primary functional areas:

    1. Equipment Management Area (Area 1): Displays the sensor list and connected devices.
    2. Control Panel Area (Area 2): Used for data stream switching, parameter configuration (exposure, white balance, etc.), image acquisition control, and D2C (Depth-to-Color) function control.
    3. Image Preview Area (Area 3): Displays real-time sensor image previews, frame rates, timestamps, and other telemetry. Clicking the log information at the bottom of this area expands the detailed logs.

    Control Panel Tabs:

    • Single Camera Mode: Standard configuration for basic camera usage.
    • Advanced Mode: Provides access to D2C, Point Cloud, and frame synchronization settings.

  4. Explore Orbbec SDK sample code

    main

    The examples/ directory contains sample code designed to demonstrate how to use the Orbbec SDK to access camera hardware and integrate it into your own applications. These examples serve as practical templates for common camera operations.

    For deep technical details, function signatures, and comprehensive API documentation, refer to the official doc section of the repository.

  5. Repository structure overview

    main

    The Orbbec SDK repository contains the following components:

    • library: Core SDK library files and C/C++ header files.
    • examples: C/C++ sample project source code.
    • doc: API reference and sample documentation.
    • driver: Windows device drivers for OpenNI protocol devices (e.g., Dabai, Astra Mini Pro, Gemini E). UVC protocol devices do not need these.
    • scripts: Linux udev rules for permission management and Windows scripts for timestamp/metadata registration.

  6. Navigate the OrbbecViewer interface

    main

    The OrbbecViewer interface is divided into three main functional areas:

    1. Device Management Area: Displays the sensor list and point cloud data.
    2. Control Panel Area: Contains switches for data streams, parameter configurations (camera, advanced mode), image capture controls, D2C (Depth-to-Color) controls, and point cloud controls.
    3. Image Preview Area: Shows real-time sensor previews, including frame rates, timestamps, and frame numbers.

    Quick Navigation Tips:

    • Language: Use the language button at the bottom left of the control panel to switch between Chinese and English.
    • Device Info/Firmware: Use the device info button to view device details or perform firmware upgrades.
    • Tabs: Use the buttons above the language toggle to switch between "Camera Control" and "Advanced Mode" tabs.

  7. Explore Orbbec SDK C++ and C Sample Code

    main

    The Orbbec SDK provides a comprehensive set of sample implementations in both C++ and C to help developers integrate camera functionality. These samples cover basic connectivity, stream acquisition (Depth, Color, IR, IMU), advanced sensor manipulation, and specialized features like point cloud generation and network mode acquisition.

    Key Capabilities by Sample Type:

    • Basic Connectivity & Info: HelloOrbbec (connect to device and retrieve SDK/device info).
    • Stream Viewers: DepthViewer, ColorViewer, InfraredViewer, and DoubleInfraredViewer (for binocular IR data on Gemini 2 XL/330 Series).
    • Data Processing: PointCloud (generate .ply files), SyncAlignViewer (sensor alignment), Post-Processing, and HdrMerge (Gemini 330 Series).
    • Device Management: HotPlugin (plug/unplug callbacks), SensorControl, FirmwareUpgrade, and MultiDevice (operating on multiple cameras simultaneously).
    • Advanced Modes: NetDevice (network-based acquisition for Femto Mega/Gemini 2 XL) and DepthWorkMode (switching depth modes on supported hardware).

  8. Monitor device connection changes with ob::Context

    main

    When performing operations that cause a device to disconnect (like a firmware upgrade or reboot), use ob::Context::setDeviceChangedCallback to listen for the device's return.

    The callback provides two lists:

    • removedList: Devices that were disconnected.
    • addedList: Devices that were newly connected.

    This is essential for managing the lifecycle of a device during a firmware upgrade cycle where the device is expected to disappear and reappear.

    ob::Context ctx;
ctx.setDeviceChangedCallback([](std::shared_ptr<ob::DeviceList> removedList, std::shared_ptr<ob::DeviceList> addedList) {
    if(addedList && addedList->deviceCount() > 0) {
        auto device = addedList->getDevice(0);
        // Check if this is the device we were waiting for
        if(deviceSN_ == std::string(device->getDeviceInfo()->serialNumber())) {
            // Handle reconnection logic
        }
    }

    if(removedList && removedList->deviceCount() > 0) {
        // Handle disconnection logic
    }
});

  9. Use Global Timestamp for multi-device synchronization

    main

    The Global Timestamp feature converts device timestamps to a system time dimension using linear regression, allowing for synchronization across multiple devices. It is primarily supported on the Gemini 330 series, though Gemini 2, 2L, Femto Mega, and Femto Bolt are supported (though stability is not fully tested).

    Key Options:

    • <GlobalTimestampFitterEnable>: Enable or disable the feature. If stability issues occur on supported devices, set this to false.
    • <GlobalTimestampFitterInterval>: Refresh interval for the fitter in milliseconds (default: 8000, min: 100, recommended max: 10000).
    • <GlobalTimestampFitterQueueSize>: Queue size for the fitter (default: 10, min: 4).
    <Misc>
    <GlobalTimestampFitterEnable>true</GlobalTimestampFitterEnable>
    <GlobalTimestampFitterInterval>8000</GlobalTimestampFitterInterval>
    <GlobalTimestampFitterQueueSize>10</GlobalTimestampFitterQueueSize>
</Misc>

  10. Configure D2C alignment for RGBD Point Clouds

    main

    To generate colored point clouds, you must align the depth stream to the color stream using D2C (Depth-to-Color) alignment. You should check for hardware support (ALIGN_D2C_HW_MODE) before falling back to software mode (ALIGN_D2C_SW_MODE).

    1. Get the color profile.
    2. Use pipeline.getD2CDepthProfileList to find a supported depth profile that matches the color profile's alignment mode.
    3. Enable the alignment mode in your configuration using config->setAlignMode().
    // 1. Configure color stream
auto colorProfiles = pipeline.getStreamProfileList(OB_SENSOR_COLOR);
if(colorProfiles) {
    auto profile = colorProfiles->getProfile(OB_PROFILE_DEFAULT);
    colorProfile = profile->as<ob::VideoStreamProfile>();
}
config->enableStream(colorProfile);

// 2. Find supported D2C depth profile
std::shared_ptr<ob::StreamProfileList> depthProfileList;
OBAlignMode alignMode = ALIGN_DISABLE;
if(colorProfile) {
    depthProfileList = pipeline.getD2CDepthProfileList(colorProfile, ALIGN_D2C_HW_MODE);
    if(depthProfileList->count() > 0) {
        alignMode = ALIGN_D2C_HW_MODE;
    } else {
        depthProfileList = pipeline.getD2CDepthProfileList(colorProfile, ALIGN_D2C_SW_MODE);
        if(depthProfileList->count() > 0) {
            alignMode = ALIGN_D2C_SW_MODE;
        }
    }
}
// ... (select depth profile and enable stream)

// 3. Enable alignment in config
config->setAlignMode(alignMode);

  11. Manage data streams with ob::Pipeline and ob::Config

    main

    The Orbbec SDK uses a Pipeline and Config pattern to manage sensor data streams:

    • ob::Pipeline: Acts as the central controller through which multiple types of streams are opened, closed, and managed. It is responsible for starting the data flow and retrieving sets of frames.
    • ob::Config: Used to define which specific streams (e.g., OB_STREAM_DEPTH) should be enabled or disabled for the pipeline.

    To use them together, you create a Config object, call enableVideoStream() for your required streams, and then pass that config to pipe.start(config).

  12. Use D2C and Point Cloud features in Advanced Mode

    main

    In Advanced Mode, you can access high-level spatial features:

    • D2C (Depth-to-Color) Alignment: Supports both hardware and software D2C modes to align depth and color data.
    • Frame Synchronization: When enabled, the SDK combines Depth and Color into a FrameSet based on hardware timestamps. This is supported on specific models like Gemini2, Gemini2 L, Astra2, Femto, Femto W, and Femto Mega.
    • 3D Point Cloud Control:
      • Use the mouse wheel to zoom and mouse movement to navigate the point cloud.
      • You can export depth point clouds (xyz) or RGBD point clouds (xyzrgb) in .ply format.
      • Exported files are saved to OrbbecViewer/output/PointCloud by default.
    • Record and Playback: Recorded files are saved to OrbbecViewer/output/RecordFile by default.