MVSplat Documentation

MVSplat uses datasets split into ~100 MB chunks. You can find small subsets of the Real Estate 10k and ACID datasets here. Unzip these into a datasets folder in the project root directory.

To convert standard versions of these datasets to the required chunked format, use the real_estate_10k_tools scripts.

To use the DTU dataset for testing, follow these steps:

1. Download the preprocessed DTU data dtu_training.rar.
2. Convert DTU to chunks using the provided script: python src/scripts/convert_dtu.py --input_dir PATH_TO_DTU --output_dir datasets/dtu
3. (Optional) Generate an evaluation index for a specific number of context views $N$: python src/scripts/generate_dtu_evaluation_index.py --n_contexts=N Note: Pre-tested versions for $N=2$ and $N=3$ are available in the /assets folder.

To install MVSplat, clone the repository, create a Conda environment with Python 3.10+, and install the specific PyTorch and requirement versions.

git clone https://github.com/donydchen/mvsplat.git
cd mvsplat
conda create -n mvsplat python=3.10
conda activate mvsplat
pip install torch==2.1.2 torchvision==0.16.2 torchaudio==2.1.2 --index-url https://download.pytorch.org/whl/cu118
pip install -r requirements.txt

Training requires a backbone pretrained weight from UniMatch.

1. Download the weight and save it to checkpoints/: wget 'https://s3.eu-central-1.amazonaws.com/avg-projects/unimatch/pretrained/gmdepth-scale1-resumeflowthings-scannet-5d9d7964.pth' -P checkpoints

2. Start training (example for re10k): python -m src.main +experiment=re10k data_loader.train.batch_size=14

Hardware Notes:

• Recommended: Single A100 (80GB) GPU.
• Multi-GPU: Can be trained on multiple GPUs with smaller RAM by reducing data_loader.train.batch_size per GPU.

# download the backbone pretrained weight from unimatch and save to 'checkpoints/'
wget 'https://s3.eu-central-1.amazonaws.com/avg-projects/unimatch/pretrained/gmdepth-scale1-resumeflowthings-scannet-5d9d7964.pth' -P checkpoints
# train mvsplat
python -m src.main +experiment=re10k data_loader.train.batch_size=14

Since MVSplat uses pytorch_lightning, you can train on multiple nodes via SLURM. For a setup with 2 nodes and 2 GPUs per node, use the following configuration in your SLURM script:

#SBATCH --nodes=2           # should match with trainer.num_nodes
#SBATCH --gres=gpu:2        # gpu per node
#SBATCH --ntasks-per-node=2

# optional, for debugging
export NCCL_DEBUG=INFO
export HYDRA_FULL_ERROR=1
# optional, set network interface, obtained from ifconfig
export NCCL_SOCKET_IFNAME=[YOUR NETWORK INTERFACE]
# optional, set IB GID index
export NCCL_IB_GID_INDEX=3

# run the command with 'srun'
srun python -m src.main +experiment=re10k \
data_loader.train.batch_size=4 \
trainer.num_nodes=2

The convert_dtu.py script converts raw DTU dataset directories into a chunked .torch format suitable for MVSplat. The script processes images and camera metadata (intrinsics and extrinsics), organizes them into chunks of approximately 100 MB, and generates a index.json file to map scan keys to their respective chunk files.

Note: This script is intended for testing only; the DTU dataset is not used for training in this pipeline.

To fine-tune using released weights without loading the optimizer states, use the following command:

python -m src.main +experiment=re10k data_loader.train.batch_size=14 \
checkpointing.load=checkpoints/re10k.ckpt \
checkpointing.resume=false

To render novel views and compute metrics, first save pretrained models to the /checkpoints directory.

For RealEstate10K (re10k):

python -m src.main +experiment=re10k \
checkpointing.load=checkpoints/re10k.ckpt \
mode=test \
dataset/view_sampler=evaluation \
test.compute_scores=true

For ACID:

python -m src.main +experiment=acid \
checkpointing.load=checkpoints/acid.ckpt \
mode=test \
dataset/view_sampler=evaluation \
dataset.view_sampler.index_path=assets/evaluation_index_acid.json \
test.compute_scores=true

Rendered views are stored in outputs/test.

To evaluate ablation models (e.g., the 'base' model), load the specific checkpoint and set the relevant model flag. For the 'base' model (which lacks depth refinement):

python -m src.main +experiment=re10k \
checkpointing.load=checkpoints/ablations/re10k_worefine.ckpt \
mode=test \
dataset/view_sampler=evaluation \
test.compute_scores=true \
wandb.name=abl/re10k_base \
model.encoder.wo_depth_refine=true 

# Table 3: base
python -m src.main +experiment=re10k \
checkpointing.load=checkpoints/ablations/re10k_worefine.ckpt \
mode=test \
dataset/view_sampler=evaluation \
test.compute_scores=true \
wandb.name=abl/re10k_base \
model.encoder.wo_depth_refine=true 

To evaluate a model trained on RealEstate10K on a different dataset like DTU, use the following command:

# Table 2: RealEstate10K -> DTU
python -m src.main +experiment=dtu \
checkpointing.load=checkpoints/re10k.ckpt \
mode=test \
dataset/view_sampler=evaluation \
dataset.view_sampler.index_path=assets/evaluation_index_dtu_nctx2.json \
test.compute_scores=true

To generate videos instead of static images, use the following command structure (example for re10k):

python -m src.main +experiment=re10k \
checkpointing.load=checkpoints/re10k.ckpt \
mode=test \
dataset/view_sampler=evaluation \
dataset.view_sampler.index_path=assets/evaluation_index_re10k_video.json \
test.save_video=true \
test.save_image=false \
test.compute_scores=false

Use save_video to save a list of FloatImage tensors as a video file. The input tensors should be in the range [0, 1]. The function uses skvideo.io.FFmpegWriter with the following settings:

• Pixel format: yuv420p
• CRF: 21
• Video filter: setpts=1.*PTS

It automatically creates parent directories if they do not exist.