DOCSEARCHbyAlgolia
Submit Docs

NVIDIA cuVSLAM

repository·main·Indexed 23 days ago

https://github.com/nvidia-isaac/cuvslam

A CUDA-accelerated library providing high-performance Visual Odometry and Simultaneous Localization and Mapping (SLAM) capabilities for various camera configurations. It includes Python bindings (PyCuVSLAM) and supports multiple tracking modes including Stereo, Stereo-Inertial, Mono-Depth (RGB-D), Monocular, and Multisensor. The library provides support for various camera distortion models (Pinhole, Fisheye, Brown, Polynomial) and integrates with datasets such as KITTI and EuRoC.

Tokens
57.1K
Snippets
143
Records
280
Agent score
82%

What's inside cuVSLAM

  2. Overview of cuVSLAM Agent Skills

    main

    The cuvslam-skills repository provides specialized agent skills for interacting with NVIDIA cuVSLAM. These skills are categorized into three main areas:

    • cuvslam-onboard: Handles environment setup, building (source, wheel, or Docker), dataset preparation (KITTI, EuRoC, TUM RGB-D, etc.), running various tracking modes (stereo, mono, multisensor, etc.), SLAM workflows (mapping, localization, loop closure), live camera setup (RealSense, ZED, OAK-D, Orbbec), and ROS 2 integration.
    • cuvslam-troubleshoot: Provides a diagnostic workflow to fix tracking or pose accuracy issues, including triage, a 14-step diagnostic process, debug data dumps for C++, Python, and ROS 2 APIs, and checks for calibration, synchronization, and image quality.
    • cuvslam-ci: Facilitates working on cuVSLAM CI/CD via GitHub Actions, covering PR verification, nightly pipelines, dataset provisioning, KPI reporting, and build/test matrix management.

  3. Use the cuvslam Python module

    main
    The cuvslam module provides Python bindings for the CUDA-accelerated Visual Odometry and Mapping library. It allows users to interface with core SLAM and Odometry functionalities, including camera calibration, IMU measurements, and pose estimation.

  4. SLAM and Advanced Feature Examples

    main

    Beyond basic odometry, cuVSLAM provides examples for:

    • SLAM: Visual mapping, localization, and saving/loading maps (e.g., KITTI).
    • Distorted Images: Handling camera distortion models (e.g., EuRoC, OAK-D, ZED).
    • Image Masking: Using static masks to prevent feature selection in specific regions or dynamic masks using PyTorch tensors.
    • PyTorch Integration: Handling PyTorch GPU tensors for real-time tasks like segmentation.
    • C++ API: High-performance implementations like EuRoC VIO and SLAM.

  5. Core classes in the cuVSLAM C++ API

    main

    The cuVSLAM C++ API is primarily built around two main classes:

    1. cuvslam::Odometry: Used for visual odometry tasks.
    2. cuvslam::Slam: Used for full SLAM (Simultaneous Localization and Mapping) capabilities.

    Depending on whether you need pure odometry or full mapping and relocalization, you should choose between these two entry points.

  6. Visual Tracking Mode Examples Overview

    main

    cuVSLAM supports various visual tracking modes depending on your sensor configuration:

    • Monocular Visual Odometry: Single camera input (e.g., EuRoC dataset).
    • Monocular-Depth Visual Odometry: Monocular camera + depth information (e.g., TUM dataset, RealSense, ZED, Orbbec).
    • Stereo Visual Odometry: Dual camera input (e.g., KITTI dataset, RealSense, ZED, OAK-D, Orbbec).
    • Stereo Visual-Inertial Odometry: Stereo cameras + IMU (e.g., EuRoC, RealSense).
    • Multi-Camera Stereo Visual Odometry: Multiple stereo pairs (e.g., Tartan Ground, R2B Galileo, RealSense).
    • Multisensor Odometry: Flexible mix of RGB, RGB-D, and optional IMU (e.g., Tartan Ground, RealSense).

    Note: All Python examples use the Rerun viewer for interactive visualization of trajectories, camera frames, and mapped features. The Rerun UI remains open after the script finishes for inspection.

  7. Understand grayscale-only internal processing

    main

    All image data is converted to grayscale before feature detection. RGB images are converted using cast_rgb2gs_cpu() (standard luminance weighting).

    Consequences:

    • Color information is discarded. If objects are only distinguishable by color and not luminance, tracking may fail.

    Tip: If one color channel is significantly noisier (e.g., the blue channel on some Bayer sensors), pre-select the cleanest channel and pass it as a mono image instead of using the automatic RGB conversion.

  8. Organize traces using Domains and DomainHelper

    main

    You can group related events and marks into a Domain. This is useful for separating different subsystems in your trace.

    Using Domain directly

    Requires managing the domain handle manually:

    auto domain = Domain("my_domain_name");

// create a mark in this domain
DomainMark(domain.get_handle(), "something_happend_inside_my_domain_name");

// also trace an event
auto ev_id = DomainTraceEventStart(domain.get_handle(), "track");
tracker->track();
DomainTraceEventEnd(domain.get_handle(), ev_id);

    Using DomainHelper (Recommended)

    DomainHelper provides a more convenient interface so you don't have to pass the handle manually for every call:

    auto helper = DomainHelper("new_domain");

helper.mark("something_happend_in_new_domain");

{
    auto ev_ = helper.trace_event("scope_event");
    foo();
}

auto ev_id_ = helper.trace_start("bar");
bar();
helper.trace_event_end(ev_id_);

  9. Select a cuVSLAM tracking mode

    main

    cuVSLAM supports several odometry modes that can be selected via Odometry::Config::odometry_mode (C++) or cuvslam.Tracker.OdometryMode (Python). The choice of mode depends on your sensor rig's capabilities and requirements for scale and robustness.

    ModeRequired sensorsOptional sensorsMode-specific settingsWhen to use
    Mono1 camera——Single-camera tracking; cheapest setup but scale-ambiguous.
    RGBD1 RGB-D camera (aligned RGB + depth)—RGBDSettingsSingle depth-aligned camera (e.g. RealSense, TUM RGB-D).
    Multicamera≥2 cameras with at least one overlapping pair (a stereo pair)up to 32 cameras total—Stereo or multi-stereo rigs. Most accurate purely-visual mode.
    Inertial1 stereo pair + 1 IMU——Stereo VIO. Adds robustness to brief visual failures.
    Multisensor≥1 RGB-D camera or ≥1 overlapping camera pair; cuNLS-enabled buildAdditional RGB/RGB-D cameras, 0 or 1 IMUMultisensorSettingsAny-mix RGB / RGB-D rigs with optional IMU, including one RGB-D camera with IMU.

    Important Notes:

    • Multisensor mode requires a cuNLS-enabled build. All other modes work with the default build.
    • IMU fusion is always on in Inertial mode and is auto-enabled in Multisensor mode when Rig::imus is non-empty.

  10. How optional inputs are resolved at the API boundary

    main

    cuVSLAM follows a pattern where optionality is resolved at the public API boundary to prevent 'optional noise' in internal functions.

    1. The public API (like Odometry::Track()) accepts optional or nullable inputs (e.g., a null Internals* pointer).
    2. The boundary layer (e.g., BuildTrackFrameSettings()) resolves these optionals into concrete, non-optional settings structs.
    3. Internal functions (e.g., IVisualOdometry::track, IMonoSOF::track) receive these concrete structs and do not need to perform has_value() checks.

    Example of the resolution flow: Internals* (null or overrides) $\rightarrow$ Internals{} (if null) $\rightarrow$ BuildTrackFrameSettings() $\rightarrow$ TrackPerFrameSettings (concrete) $\rightarrow$ Internal tracking functions (no optionals).

  11. Avoid using setters for per-frame parameter changes

    main

    Do not use setter methods (e.g., odometry.SetNumDesiredTracks(200)) to change a parameter for a single frame. Setters mutate the long-lived state of the Odometry object, which can cause parameter changes to 'bleed' into subsequent frames, making behavior difficult to reproduce and test.

    Incorrect Pattern:

    // BAD — setter mutates shared state and can bleed across frames.
odometry.SetNumDesiredTracks(200);
odometry.Track(images);

  12. How the cuVSLAM system architecture works

    main

    cuVSLAM operates using a decoupled architecture that separates real-time tracking from non-real-time SLAM processing (mapping and localization) via a thread-safe message queue.

    Core Components:

    • Real-time (RT) SLAM: Handles high-frequency tasks including Track (real-time odometry), Localize (finding position in the map), and SaveMap.
    • ThreadSafeMessageQueue: Acts as the intermediary between the real-time tracking thread and the background processing thread. It handles messages such as AddKF (adding a keyframe), FindLC (finding loop closure), and SaveMap requests.
    • Background (BG) SLAM: A non-real-time thread that continuously processes messages from the queue to update the global map.
    • ThreadSafe Tail (Keyframe Chain): A specialized data structure that allows the real-time tracking thread to interact with the background thread without blocking, facilitating continuous pose estimation even while the map is being updated.