DOCSEARCHbyAlgolia
Submit Docs

BundleSDF Documentation

repository·master·Indexed 23 days ago

https://github.com/nvlabs/bundlesdf

BundleSDF is a neural 6-DoF tracking and 3D reconstruction system for unknown rigid objects using monocular RGBD video. It utilizes a Neural Object Field and pose graph optimization to handle occlusions, specular highlights, and untextured surfaces. The system includes integration with LoFTR for feature matching and XMem for segmentation, providing tools for joint tracking, global refinement, and benchmarking on the HO3D dataset.

Tokens
4.5K
Snippets
14
Records
18
Agent score
80%

What's inside BundleSDF

  1. Build dataset symlinks for LoFTR

    master

    To organize the data, symlink your downloaded datasets into the data directory under the main LoFTR project directory.

    ScanNet Symlinks

    Link the training/test sets and the indices:

    ln -s /path/to/scannet_train/* /path/to/LoFTR/data/scannet/train
ln -s /path/to/scannet_test/* /path/to/LoFTR/data/scannet/test
ln -s /path/to/scannet_indices/* /path/to/LoFTR/data/scannet/index

    MegaDepth Symlinks

    Note that for MegaDepth, the training and test sets share the same dataset. Use -sv to link the specific directories:

    ln -sv /path/to/megadepth/phoenix /path/to/megadepth_d2net/Undistorted_SfM /path/to/LoFTR/data/megadepth/train
ln -sv /path/to/megadepth/phoenix /path/to/megadepth_d2net/Undistorted_SfM /path/to/LoFTR/data/megadepth/test
ln -s /path/to/megadepth_indices/* /path/to/LoFTR/data/megadepth/index
    # scannet
ln -s /path/to/scannet_train/* /path/to/LoFTR/data/scannet/train
ln -s /path/to/scannet_test/* /path/to/LoFTR/data/scannet/test
ln -s /path/to/scannet_indices/* /path/to/LoFTR/data/scannet/index

# megadepth
ln -sv /path/to/megadepth/phoenix /path/to/megadepth_d2net/Undistorted_SfM /path/to/LoFTR/data/megadepth/train
ln -sv /path/to/megadepth/phoenix /path/to/megadepth_d2net/Undistorted_SfM /path/to/LoFTR/data/megadepth/test
ln -s /path/to/megadepth_indices/* /path/to/LoFTR/data/megadepth/index

  2. Run and Benchmark on HO3D Dataset

    master

    To evaluate BundleSDF on the HO3D dataset, use the following commands:

    1. Generate Results:

      python run_ho3d.py --video_dirs <HO3D_VIDEO_DIRS> --out_dir <OUT_DIR>

    2. Benchmark Results:

      python benchmark_ho3d.py --video_dirs <HO3D_VIDEO_DIRS> --out_dir <OUT_DIR>
    # Run BundleSDF to get the pose and reconstruction results
python run_ho3d.py --video_dirs /mnt/9a72c439-d0a7-45e8-8d20-d7a235d02763/DATASET/HO3D_v3/evaluation/SM1 --out_dir /home/bowen/debug/ho3d_ours

# Benchmark the output results
python benchmark_ho3d.py --video_dirs /mnt/9a72c439-d0a7-45e8-8d20-d7a235d02763/DATASET/HO3D_v3/evaluation/SM1 --out_dir /home/bowen/debug/ho3d_ours

  3. Extract LoFTR dataset indices

    master

    Download the required dataset indices from the provided link and unzip the files. You can then extract the indices and optional testing data using tar.

    unzip downloaded-file.zip

# extract dataset indices
tar xf train-data/megadepth_indices.tar
tar xf train-data/scannet_indices.tar

# extract testing data (optional)
tar xf testdata/megadepth_test_1500.tar
tar xf testdata/scannet_test_1500.tar
    unzip downloaded-file.zip

# extract dataset indices
tar xf train-data/megadepth_indices.tar
tar xf train-data/scannet_indices.tar

# extract testing data (optional)
tar xf testdata/megadepth_test_1500.tar
tar xf testdata/scannet_test_1500.tar

  4. Prepare Custom RGBD Data for Processing

    master

    To run BundleSDF on your own data, organize your RGBD video folder with the following structure:

    root
  ├──rgb/    (PNG files)
  ├──depth/  (PNG files, PNG format, uint16, stored in mm. Filename must match RGB file)
  ├──masks/  (PNG files. Filename must match RGB file. 0 is background, non-zero is foreground)
  └──cam_K.txt   (3x3 intrinsic matrix, space or enter delimited)
    root
  ├──rgb/    (PNG files)
  ├──depth/  (PNG files, stored in mm, uint16 format. Filename same as rgb)
  ├──masks/       (PNG files. Filename same as rgb. 0 is background. Else is foreground)
  └──cam_K.txt   (3x3 intrinsic matrix, use space and enter to delimit)

  5. Install and Setup BundleSDF via Docker

    master

    BundleSDF uses Docker for environment setup. Follow these steps to build the image and compile machine-dependent packages:

    1. Build the Docker image (one-time setup):

      cd docker
docker build --network host -t nvcr.io/nvidian/bundlesdf .

    2. Start the container and compile packages:

      cd docker && bash run_container.sh
# Inside the container:
bash build.sh
    cd docker
docker build --network host -t nvcr.io/nvidian/bundlesdf .

cd docker && bash run_container.sh
# Inside docker container:
bash build.sh

  6. Download Required Pretrained Weights

    master

    To run the system, you must manually download and place pretrained weights in the following locations:

    • Segmentation Network (XMem): Download the weights and place them at ./BundleTrack/XMem/saves/XMem-s012.pth.
    • LoFTR (outdoor_ds): Download the weights and place them at ./BundleTrack/LoFTR/weights/outdoor_ds.ckpt.

  8. Prepare MegaDepth and ScanNet datasets for LoFTR training

    master

    Training LoFTR requires two components: the original datasets (ScanNet and MegaDepth) and offline generated dataset indices.

    MegaDepth Setup

    You need two parts of the MegaDepth data:

    1. Original Depth Maps: Download from the original MegaDepth dataset. This is referred to as /path/to/megadepth.
    2. D2-Net Preprocessed Images: Download D2-Net preprocessed images. These are manually undistorted to ensure compatibility with intrinsics. This is referred to as /path/to/megadepth_d2net.

    ScanNet Setup

    Follow the official ScanNet guide. Important: Use the python exported data instead of the C++ exported version.

  9. Train LoFTR on ScanNet or MegaDepth

    master

    LoFTR provides training scripts for both ScanNet and MegaDepth.

    Matcher Types:

    • Scripts ending in _ot use the optimal-transport matcher.
    • Scripts ending in _ds use the dual-softmax matcher.

    Note on Reproduction: The provided scripts are scaled down for smaller setups (e.g., 4 GPUs). To reproduce the exact results from the LoFTR paper, you may need to scale the setup (e.g., using 32 GPUs) and linearly scale the learning rate and warm-up.

    Training Commands

    ScanNet (Dual-Softmax):

    scripts/reproduce_train/indoor_ds.sh

    MegaDepth (Dual-Softmax):

    scripts/reproduce_train/outdoor_ds.sh

    Note: The MegaDepth script uses smaller image sizes of 640x640.

    scripts/reproduce_train/indoor_ds.sh

scripts/reproduce_train/outdoor_ds.sh

  10. Reproduce testing results

    master

    To reproduce testing results for ScanNet or MegaDepth, first set up symlinks from your downloaded datasets to the data/ directory in the LoFTR project:

    ln -s /path/to/scannet-1500-testset/* /path/to/LoFTR/data/scannet/test
ln -s /path/to/megadepth-1500-testset/* /path/to/LoFTR/data/megadepth/test

    Then, you can run the evaluation using the provided shell scripts or directly via test.py with pytorch-lightning arguments.

  11. Install LoFTR

    master

    You can install LoFTR in two ways depending on your requirements:

    1. Full Installation (Recommended): Includes pytorch-lightning trainer features. Use Conda to create the environment from the provided YAML file.
    2. Matcher Only: A lightweight installation for using only the LoFTR matcher.

    Note for LoFTR-OT users: If you intend to use the LoFTR-OT (Optimal Transport) model, you must manually download superglue.py from the SuperGlue repository due to licensing restrictions.

    # For full pytorch-lightning trainer features (recommended)
conda env create -f environment.yaml
conda activate loftr

# For the LoFTR matcher only
pip install torch einops yacs kornia

  12. Run 6-DoF Tracking and Reconstruction on Custom Data

    master

    Processing custom data involves three sequential steps using run_custom.py.

    Note on Depth Scale: The default configuration assumes max relevant depth < 1. If your data uses different scales, update BundleTrack/config_ho3d.yml at the relevant line.

    1. Joint Tracking and Reconstruction: Runs the core pipeline. Use --use_segmenter 1 if you have provided masks or are using the segmenter.

      python run_custom.py --mode run_video --video_dir <VIDEO_DIR> --out_folder <OUT_FOLDER> --use_segmenter 1 --use_gui 1 --debug_level 2

    2. Global Refinement: Post-processing step to refine the reconstructed mesh.

      python run_custom.py --mode global_refine --video_dir <VIDEO_DIR> --out_folder <OUT_FOLDER>

    3. Visualize Pose (Optional): Draws the oriented bounding box to visualize the tracked pose.

      python run_custom.py --mode draw_pose --out_folder <OUT_FOLDER>

    Outputs: Results are saved in <OUT_FOLDER>, including tracked poses in ob_in_cam/ and the reconstructed mesh in textured_mesh.obj.

    # 1) Run joint tracking and reconstruction. 
python run_custom.py --mode run_video --video_dir /home/bowen/debug/2022-11-18-15-10-24_milk --out_folder /home/bowen/debug/bundlesdf_2022-11-18-15-10-24_milk --use_segmenter 1 --use_gui 1 --debug_level 2

# 2) Run global refinement post-processing to refine the mesh
python run_custom.py --mode global_refine --video_dir /home/bowen/debug/2022-11-18-15-10-24_milk --out_folder /home/bowen/debug/bundlesdf_2022-11-18-15-10-24_milk

# 3) (Optional) Draw oriented bounding box
python run_custom.py --mode draw_pose --out_folder /home/bowen/debug/bundlesdf_2022-11-18-15-10-24_milk