repository·main·Indexed 12 days ago
https://github.com/facebookresearch/vggtA feed-forward neural network designed to infer 3D scene attributes—including camera parameters, depth maps, point maps, and 3D point tracks—from one or more image views in seconds. It supports zero-shot single-view reconstruction and can export results to COLMAP format for integration with tools like gsplat.
Standard model checkpoints may have different licensing terms. Only the specific checkpoint hosted on Hugging Face allows for commercial usage. This commercial checkpoint (VGGT-1B-Commercial) achieves performance levels comparable to or slightly better than the original (e.g., AUC@30: 90.37 vs 89.98 on the Co3D dataset).
https://huggingface.co/facebook/VGGT-1B-CommercialTo install VGGT, clone the repository and install the required dependencies using pip.
Required dependencies include torch, torchvision, numpy, Pillow, and huggingface_hub.
git clone git@github.com:facebookresearch/vggt.git
cd vggt
pip install -r requirements.txtThe COLMAP files exported by VGGT can be used directly for training with gsplat.
Workflow:
demo_colmap.py.gsplat (version 1.3.0 is recommended).cd gsplat
python examples/simple_trainer.py default --data_factor 1 --data_dir /YOUR/SCENE_DIR/ --result_dir /YOUR/RESULT_DIR/Before installing VGGT, you must manually install torch and torchvision to prevent CUDA version mismatches. It is recommended to install PyTorch 2.3.1 with CUDA 12.1.
# install pytorch 2.3.1 with cuda 12.1
pip install torch==2.3.1 torchvision==0.18.1 --index-url https://download.pytorch.org/whl/cu121You can run VGGT inference with a few lines of code. The model automatically downloads pretrained weights from Hugging Face on the first run.
Note on Precision: bfloat16 is supported on Ampere GPUs (Compute Capability 8.0+). For older GPUs, use float16.
import torch
from vggt.models.vggt import VGGT
from vggt.utils.load_fn import load_and_preprocess_images
device = "cuda" if torch.cuda.is_available() else "cpu"
# bfloat16 is supported on Ampere GPUs (Compute Capability 8.0+)
dtype = torch.bfloat16 if torch.cuda.get_device_capability()[0] >= 8 else torch.float16
# Initialize the model and load the pretrained weights.
model = VGGT.from_pretrained("facebook/VGGT-1B").to(device)
# Load and preprocess example images (replace with your own image paths)
image_names = ["path/to/imageA.png", "path/to/imageB.png", "path/to/imageC.png"]
images = load_and_preprocess_images(image_names).to(device)
with torch.no_grad():
with torch.cuda.amp.autocast(dtype=dtype):
# Predict attributes including cameras, depth maps, and point maps.
predictions = model(images)The effective batch size is calculated as batch_size_per_gpu * num_gpus. Because the learning rate depends on this effective batch size, it is recommended to test several values.
Suggested starting values to try:
5e-61e-55e-51e-45e-4To set up the VGGT training environment, follow these steps:
Install the package in editable mode:
pip install -e .Prepare Datasets:
pip install -e .To use the interactive visualization tools, first install the demo dependencies:
pip install -r requirements_demo.txtLaunch a local web interface to upload images/videos and interactively explore the 3D scene:
python demo_gradio.pyRun reconstruction and visualize point clouds in viser. This script requires a folder containing only image files:
python demo_viser.py --image_folder path/to/your/images/folderUse the --use_point_map flag to use the point cloud from the point map branch instead of the depth-based reconstruction.
pip install -r requirements_demo.txt
# Gradio
python demo_gradio.py
# Viser
python demo_viser.py --image_folder path/to/your/images/folderUse uv to run the Gradio demo, ensuring the demo extra is included.
uv run --extra demo demo_gradio.pyUse pixi to create a reproducible environment and run the Gradio demo.
pixi run -e python demo_gradio.py