DOCSEARCHbyAlgolia
Submit Docs

SplaTAM (Splat, Track & Map)

repository·main·Indexed 24 days ago

https://github.com/spla-tam/splatam

A dense RGB-D SLAM system utilizing 3D Gaussian Splatting for simultaneous tracking and mapping to achieve high-quality 3D reconstruction. It supports online demos via the NeRFCapture app, offline reconstruction, and integration with datasets such as Replica, TUM-RGBD, ScanNet, and ScanNet++. The system provides tools for interactive visualization, .ply export, and post-SLAM optimization.

Tokens
1.6K
Snippets
5
Records
10
Agent score
31%

What's inside SplaTAM

  2. Install SplaTAM using Docker

    main

    You can use the provided Docker image. It is recommended to use a virtual environment (venv) inside the container to manage dependencies.

    docker pull nkeetha/splatam:v1
bash bash_scripts/start_docker.bash
cd /SplaTAM/
pip install virtualenv --user
mkdir venv
cd venv
virtualenv --system-site-packages splatam
source ./splatam/bin/activate
pip install -r venv_requirements.txt

  3. Install SplaTAM using Singularity

    main

    Setting up a Singularity container is similar to the Docker setup. Pull the image, start an instance with NVIDIA support, and then set up a virtual environment inside the project directory.

    cd </path/to/singularity/folder/>
singularity pull splatam.sif docker://nkeetha/splatam:v1
singularity instance start --nv splatam.sif splatam
singularity run --nv instance://splatam
cd <path/to/SplaTAM/>
pip install virtualenv --user
mkdir venv
cd venv
virtualenv --system-site-packages splatam
source ./splatam/bin/activate
pip install -r venv_requirements.txt

  4. Core SplaTAM Usage Commands

    main

    Use these commands to run the SLAM pipeline, visualize results, or export data. These examples use the configs/iphone/ path as a template; replace it with your specific dataset config.

    TaskCommand
    Run SLAMpython scripts/splatam.py <config_path>
    Final Interactive Visualizationpython viz_scripts/final_recon.py <config_path>
    Online Visualizationpython viz_scripts/online_recon.py <config_path>
    Export Splats to .plypython scripts/export_ply.py <config_path>
    Post-SLAM Optimizationpython scripts/post_splatam_opt.py <config_path>
    GS on Ground Truth Posespython scripts/gaussian_splatting.py <config_path>

  5. Download Datasets

    main

    SplaTAM supports several datasets. By default, DATAROOT is set to ./data. You may need to update the input_folder in your scene-specific config files if your data is stored elsewhere.

    Replica:

    bash bash_scripts/download_replica.sh

    TUM-RGBD:

    bash bash_scripts/download_tum.sh

    ScanNet: Follow the ScanNet website for downloading. You must extract color/depth frames from .sens files using a SensReader script. The expected directory structure is:

    DATAROOT
└── scannet
      └── scene0000_00
            └── frames
                  ├── color
                  ├── depth
                  ├── intrinsic
                  └── pose

    ScanNet++: Follow the ScanNet++ website for downloading and undistorting. Note that for sequence b20a261fdf, only the first 360 frames are used due to trajectory jumps.

  6. Run an Online Demo with NeRFCapture

    main

    You can use an iPhone or LiDAR-equipped Apple device with the NeRFCapture app to perform real-time SLAM. Ensure both the iPhone and PC are on the same WiFi network.

    1. Run the online demo script on your PC:
    bash bash_scripts/online_demo.bash configs/iphone/online_demo.py
    1. In the app, tap 'send' successively for each frame. The app will disconnect once capturing is complete, and the PC will display the interactive reconstruction.

  7. Run SplaTAM Offline or Collect Datasets

    main

    If you have already captured data using the NeRFCapture app, you can process it offline or convert it into a standard dataset format.

    Run offline reconstruction:

    bash bash_scripts/nerfcapture.bash configs/iphone/nerfcapture.py

    Convert NeRFCapture data to a dataset:

    bash bash_scripts/nerfcapture2dataset.bash configs/iphone/dataset.py

  8. Install SplaTAM via Conda

    main

    The recommended installation method uses Anaconda and pip. SplaTAM is benchmarked with Python 3.10, Torch 1.12.1, and CUDA 11.6, though it supports newer versions like Torch 2.3.0 and CUDA 12.1.

    Follow these steps to create a dedicated environment and install dependencies:

    conda create -n splatam python=3.10
conda activate splatam
conda install -c "nvidia/label/cuda-11.6.0" cuda-toolkit
conda install pytorch==1.12.1 torchvision==0.13.1 torchaudio==0.12.1 cudatoolkit=11.6 -c pytorch -c conda-forge
pip install -r requirements.txt

  9. Configure Benchmarking with Weights & Biases

    main

    For logging during benchmarking, SplaTAM supports Weights & Biases (wandb).

    To enable it:

    1. Set the wandb flag to True in your configuration file.
    2. Specify the wandb_folder path.
    3. Update the entity config to match your wandb account name.

    Ensure you have a wandb account created before running.

  10. Cite SplaTAM in research

    main

    If you use SplaTAM in your research, please cite the following paper:

    @inproceedings{keetha2024splatam,
        title={SplaTAM: Splat, Track & Map 3D Gaussians for Dense RGB-D SLAM},
        author={Keetha, Nikhil and Karhade, Jay and Jatavallabhula, Krishna Murthy and Yang, Gengshan and Scherer, Sebastian and Ramanan, Deva and Luiten, Jonathon},
        booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
        year={2024}
      }