DOCSEARCHbyAlgolia
Submit Docs

VGGSfM (Visual Geometry Grounded Deep Structure From Motion)

repository·main·Indexed 23 days ago

https://github.com/facebookresearch/vggsfm

A framework for 3D reconstruction and camera pose estimation designed for sequential video frames. VGGSfM recovers camera poses and point clouds, supporting dynamic sequence handling via masks, dense depth prediction using Depth-Anything-V2, and sliding window reconstruction for large video sequences. It outputs results in COLMAP format, ensuring compatibility with NeRF and Gaussian Splatting codebases.

Tokens
1.7K
Snippets
6
Records
12
Agent score
31%

What's inside VGGSfM

  1. Filter Dynamic Objects with Masks

    main

    To handle moving objects that might interfere with reconstruction, you can provide binary masks.

    Setup:

    • Place masks in SCENE_DIR/masks/.
    • Filenames must match the image filenames (e.g., images/0000.png $\rightarrow$ masks/0000.png).
    • Mask Values: 1 indicates pixels to be filtered out (dynamic); 0 indicates pixels to keep.

  2. Use Custom Image Data

    main

    To use your own dataset, provide the directory containing your images via SCENE_DIR. The images must be located in a subfolder named images/ within that directory.

    Example structure:

    YOUR_FOLDER/
└── images/
    ├── 0000.png
    └── ...
    python demo.py SCENE_DIR=/YOUR_FOLDER camera_type=SIMPLE_RADIAL gr_visualize=True make_reproj_video=True

  3. Dense Depth Prediction (Beta)

    main

    VGGSfM can extract dense depth maps by aligning Depth-Anything-V2 predictions with the sparse SfM point cloud.

    Prerequisites:

    1. Install scikit-learn: pip install scikit-learn
    2. Clone and install Depth-Anything-V2:
      git clone git@github.com:DepthAnything/Depth-Anything-V2.git dependency/depth_any_v2
python -m pip install -e dependency/depth_any_v2

    Usage: Set dense_depth=True in demo.py. Depth maps are saved in SCENE_DIR/depths in COLMAP format. To visualize in Visdom, use visual_dense_point_cloud=True.

  4. Train Gaussian Splatting from VGGSfM Output

    main

    The sparse reconstruction output is compatible with gsplat.

    Workflow:

    1. Ensure SCENE_DIR/sparse contains cameras.bin, images.bin, and points3D.bin.
    2. Install gsplat==1.3.0.
    3. Run the trainer pointing to your SCENE_DIR.
    cd gsplat
python examples/simple_trainer.py default --data_factor 1 --data_dir /YOUR/SCENE_DIR/ --result_dir /YOUR/RESULT_DIR/

  5. Reconstruct Large Video Sequences

    main

    For sequences with over 1,000 frames (e.g., video), use video_demo.py which implements a sliding window reconstruction approach.

    Requirements:

    • Images must be in YOUR_VIDEO_FOLDER/images and named sequentially (e.g., 0000.png, 0001.png).

    Configuration:

    • init_window_size and window_size: Control the number of frames per window.
    • joint_BA_interval: Controls the frequency of joint bundle adjustment across the sequence.
    • Other flags (visualization, output) are identical to demo.py.
    python video_demo.py SCENE_DIR=/YOUR_VIDEO_FOLDER

  6. Install VGGSfM

    main

    Install VGGSfM using the provided installation script. By default, this sets up a conda environment with Python 3.10, PyTorch 2.1, and CUDA 12.1. The script also installs dependencies including pytorch3d, lightglue, pycolmap, poselib, and visdom.

    Note: If you cannot install pytorch3d on your machine, you can skip it. pytorch3d is currently only used for visdom visualization (when cfg.viz_visualize=True).

    source install.sh
python -m pip install -e .

  7. Visualize Reconstruction Results

    main

    VGGSfM provides several ways to visualize the output:

    1. Gradio (Recommended): Best for remote servers. Set gr_visualize=True to generate a web link.
    2. Visdom: Requires starting a server first via visdom command. Enable with viz_visualize=True.
    3. 2D Reprojections: Set make_reproj_video=True to generate reproj.mp4 in SCENE_DIR/visuals.
    4. Track Predictions: Set visual_tracks=True to generate track.mp4 showing point visibility/confidence.

  8. Download the VGGSfM Pre-trained Model

    main

    The VGGSfM checkpoint is automatically downloaded from Hugging Face during the first run. If you need to download it manually or specify a custom path, you can find the weights on Hugging Face or Google Drive.

    To use a manual path, set auto_download_ckpt=False and update resume_ckpt to your local path in the Hydra configuration.

  10. Generate a Denser Point Cloud

    main

    To increase point density without full optimization, you can triangulate additional points using a pixel interval grid.

    Set extra_pt_pixel_interval to define the sampling grid. Use concat_extra_points=True to append these points to the existing cloud (useful for Gaussian Splatting). The extra points are saved in additional/additional_points_dict.pt.

    python demo.py extra_pt_pixel_interval=10 concat_extra_points=True

  11. Run a 3D Reconstruction Demo

    main

    You can run the reconstruction using demo.py. The reconstruction results (camera parameters and 3D points) are saved in COLMAP format (cameras.bin, images.bin, and points3D.bin) under SCENE_DIR/sparse. This format is compatible with NeRF and Gaussian Splatting codebases.

    Common usage patterns:

    • Default settings: python demo.py SCENE_DIR=examples/kitchen
    • Change query method: Use query_method with values like sp, sift, aliked, or combinations like sp+sift.
    • Increase precision: Increase max_query_pts (default is 2048).
    • Fast reconstruction: Set fine_tracking=False to use coarse matching only.
    • Shared camera model: If images are from a video where focal length is constant, set shared_camera=True and consider camera_type=SIMPLE_RADIAL.
    # Use default settings
python demo.py SCENE_DIR=examples/kitchen 

# Specify query method: sp+sift (default: aliked)
python demo.py SCENE_DIR=examples/statue query_method=sp+sift

# Increase query number to 4096 (default: 2048)
python demo.py SCENE_DIR=examples/british_museum max_query_pts=4096 

# Assume a shared camera model and use SIMPLE_RADIAL
python demo.py shared_camera=True camera_type=SIMPLE_RADIAL query_frame_num=6

# Fast reconstruction without fine tracking
python demo.py SCENE_DIR=examples/kitchen fine_tracking=False

  12. Troubleshoot Out-of-Memory (OOM) Errors

    main

    If you encounter OOM errors due to high frame counts or query points, you can manually adjust the chunking hyperparameters in the source code. These are tuned for 32GB GPUs:

    • max_points_num: Located in vggsfm/runners/runner.py.
    • max_tri_points_num: Located in vggsfm/utils/triangulation.py.

    Reduce these values to fit your GPU's memory capacity.