OKVIS2-X Multi-Sensor SLAM System

To use the Replica Dataset in a stereo camera setup for FindAnything, you must create a specific Conda environment and download the necessary datasets. This process involves setting up habitat-sim and acquiring both the standard Replica dataset and the iMAP Replica sequences.

conda create -n replica-imap-stereo python=3.8 cmake=3.14
conda activate replica-imap-stereo
conda install habitat-sim=0.2.2 -c conda-forge -c aihabitat

The evaluation requires several external packages and specific files. Use the provided setup_eval_dependencies.sh script to download the tested commits of required packages.

Key Dependencies:

• ConceptGraphs: Re-uses evaluation setup.
• ChamferDist: Install via its setup.py.
• GradSLAM utilities: Uses pointclouds.py, projutils.py, and structutils.py from the GradSLAM package.
• Requirements: Install via the provided requirements.txt.

Note: You must unpack text_queries.zip onto the same level as the setup script.

To build the project without ROS2, clone the repository with submodules and use CMake.

Important CMake Flags:

• -DUSE_COLIDMAP=ON: Required to run FindAnything.
• -DHAVE_LIBREALSENSE=OFF: Use if you do not have a RealSense camera.
• -DUSE_NN=OFF: Use if you do not want to use LibTorch/Neural Networks.
• -DUSE_GPU=ON: Enable NVIDIA GPU for inference.

If you want to run FindAnything, you must set -DUSE_COLIDMAP=ON and -DUSE_NN=ON.

git clone --recurse-submodules git@github.com:ethz-mrl/OKVIS2-X.git

# If you forgot submodules:
git submodule update --init --recursive

mkdir build && cd build
cmake -DCMAKE_BUILD_TYPE=Release -DUSE_COLIDMAP=ON -DUSE_NN=ON ..
make -j$(nproc)

Execute the semantic accuracy evaluation using replica_eval_sem.py. This script compares FindAnything's generated outputs against ground truth point clouds and trajectories using CLIP embeddings for queries.

python3 replica_eval_sem.py --replica_seq_path /path/to/generated/sequence/ --gt_path /path/to/gt_pcl_and_traj/ --results_path /path/to/results_base/ --query_path ./text_queries/

After setting up the environment and downloading the Replica and iMAP datasets, use the replica-imap-stereo.py script to render a second image with a 6cm baseline to create a stereo setup.

Dataset Requirements:

• Replica Dataset: Download using the Facebook Research download script.
• iMAP Replica sequences: Download from this link.

Usage: replica-imap-stereo.py REPLICA_DIR IMAP_DIR OUT_DIR [SCENE] ...

Run with -h for detailed help.

To fuse global position measurements (GNSS), add a gps_parameters block to your configuration file. Supported data_type values are cartesian or geodetic.

Parameters:

• data_type: cartesian or geodetic.
• r_SA: Antenna {A} offset in IMU {S} frame (3D vector).
• yaw_error_threshold: Threshold on yaw observability to freeze optimization of extrinsics.
• robust_gps_init: Boolean to use a robustified version of Initialization.

Note on Leap Seconds: The GNSS_LEAP_NANOSECONDS variable in DatasetReaderBase can be used to account for GNSS clock offsets (default is 0; current value is 18e+9).

gps_parameters:
  data_type: cartesian # cartesian or geodetic
  r_SA: [0.04943289490451834, 0.014787790366790175, 0.6079887122447304]
  yaw_error_threshold: 0.1
  robust_gps_init: false

To enable submap alignment factors, set use_map_to_map_factors or use_map_to_live_factors in the mapping config file.

LiDAR Configuration: When running with LiDAR and submap alignment, add a lidar block to the estimator config:

lidar:
  elevation_resolution_angle: 1.0 
  azimuth_resolution_angle: 0.18
  T_SL: [ ... ] # Extrinsics IMU {S} <-> LiDAR {L}

Depth Network Configuration: When using depth networks, set parameters in the okvis2-config file:

cameras:
  - ...
    mapping: true
    mapping_rectification: true
camera_parameters:
  deep_stereo_indices: [0,1]
  fov_scale: 1.0

When using the library in client code, you must maintain an okvis::Trajectory object by implementing two callbacks to ensure a consistent trajectory that can be queried at any time.

1. Optimization Callback: Register with okvis::ViInterface::setOptimisedGraphCallback. This is called when the estimator updates (e.g., during loop closure). Use okvis::Trajectory::update to apply these updates.
2. IMU Callback: Register with okvis::ViSensorBase::setImuCallback. This provides high-rate state updates via prediction. Use okvis::Trajectory::addImuMeasurement to update the trajectory.

Coordinate Frames:

• W: OKVIS World frame (z up)
• C_i: i-th camera frame
• S: IMU sensor frame
• B: User-specified/Robot body frame

Arguments for the replica_eval_sem.py evaluation script.

--replica_seq_path: Path to generated Dataset
--gt-path: Path to Ground Truth Point Clouds and Trajectory
--results_path: Base Folder of FindAnything generated Outputs
--query_path: Path where Queries (CLIP embedding of category as a txt file) are stored. For replica we provide them in `text_queries.zip`
--n_exclude: Number of classes to be excluded (following ConceptGraph logic, default: 6)
--debug: Runs some Open3D based visualization of predicted class for debugging (Ground Truth / Predicted can be toggled by pressing `1`/`2`)