DOCSEARCHbyAlgolia
Submit Docs

DepthSplat

repository·main·Indexed 22 days ago

https://github.com/cvg/depthsplat

A framework bridging Gaussian Splatting and depth estimation to improve novel view synthesis and enable unsupervised depth pre-training. It supports cross-task interactions to reduce depth prediction error and enhance rendering results. The system requires Python 3.10, PyTorch 2.4.0, and CUDA 12.4, and provides pre-trained models via Hugging Face.

Tokens
10.6K
Snippets
8
Records
62
Agent score
78%

What's inside DepthSplat

  1. Overview of DepthSplat

    main

    DepthSplat is a framework designed to enable cross-task interactions between Gaussian Splatting and depth estimation. It facilitates two primary workflows:

    1. Improved Novel View Synthesis: Using better depth information to enhance Gaussian splatting results.
    2. Unsupervised Depth Pre-training: Using Gaussian splatting to reduce depth prediction error through unsupervised pre-training.

  2. Organize dataset folder structure

    main

    The codebase expects datasets to be organized into a specific directory structure containing .torch chunk files and index.json files for both train and test splits.

    Expected structure:

    ├── datasets
│   ├── re10k
│   │   ├── train
│   │   │   ├── 000000.torch
│   │   │   ├── ...
│   │   │   ├── index.json
│   │   ├── test
│   │   │   ├── 000000.torch
│   │   │   ├── ...
│   │   │   ├── index.json
│   │   ├── dl3dv
│   │   │   ├── train
│   │   │   │   ├── 000000.torch
│   │   │   │   ├── ...
│   │   │   │   ├── index.json
│   │   │   ├── test
│   │   │   │   ├── 000000.torch
│   │   │   │   ├── ...
│   │   │   │   ├── index.json

  3. Understand DepthSplat Camera Conventions

    main

    When providing camera data to DepthSplat, ensure it adheres to the following conventions:

    • Intrinsics: Camera intrinsic matrices must be normalized. The first row should be divided by the image width, and the second row should be divided by the image height.
    • Extrinsics: Camera extrinsic matrices follow the OpenCV convention for camera-to-world transformation (+X right, +Y down, +Z pointing into the screen).

  4. Prepare RealEstate10K dataset

    main

    To use RealEstate10K, you must convert the original dataset into .torch chunks.

    • 360p (360x640): Follow the acquisition instructions in the pixelSplat repo.
    • 720p (720x1280):
      1. Download the 720p version using a downloading script (modify the script to change 360p to 720p in the source code).
      2. Use the real_estate_10k_tools to convert the downloaded data into the required format.

    Quick Start: For rapid inference testing, you can download a preprocessed 720p test subset containing two scenes from Hugging Face.

  5. Prepare DL3DV dataset

    main

    DL3DV processing involves converting raw data into .torch chunks and generating index files.

    Test Set (DL3DV-Benchmark)

    1. Run src/scripts/convert_dl3dv_test.py to convert the test set.
    2. Run src/scripts/generate_dl3dv_index.py to create the index.json file.

    Training Set (DL3DV-480p)

    1. Run src/scripts/convert_dl3dv_train.py to convert the training set.
    2. Run src/scripts/generate_dl3dv_index.py to create the index.json file.

    High-Resolution (DL3DV-960P)

    Follow the same procedure as the 480p version, but ensure you update the images_8 folder to images_4 in the processing scripts.

    Note: You must update the dataset paths within the processing scripts before running them.

    Quick Start: A preprocessed 960p test subset is available for research purposes on Hugging Face.

  7. Setup for Training DepthSplat

    main

    Before starting training, you must download specific pre-trained weights and configure your Weights & Biases (wandb) account.

    1. Download Weights

    Download the required UniMatch and Depth Anything V2 weights into the pretrained/ directory:

    wget https://s3.eu-central-1.amazonaws.com/avg-projects/unimatch/pretrained/gmflow-scale1-things-e9887eda.pth -P pretrained
wget https://huggingface.co/depth-anything/Depth-Anything-V2-Small/resolve/main/depth_anything_v2_vits.pth -P pretrained

    2. Configure Logging

    Set up your wandb account and update config/main.yaml by setting wandb.entity=YOUR_ACCOUNT to enable logging.

    3. Hardware Requirements

    While the authors use four GH200 GPUs, you can train on other configurations (e.g., four RTX 4090s or a single A100 80GB). To maintain consistent results, ensure the total number of training samples remains constant:

    $$ ext{Total Samples} = ext{num GPUs} imes ext{data_loader.train.batch_size} imes ext{trainer.max_steps}$$

  8. Set up the pretrained models directory

    main

    The project expects pre-trained weights to be stored in a directory named pretrained. To manage your model files, it is recommended to download your weights to a specific location (YOUR_MODEL_PATH) and then create a symbolic link to the pretrained directory in your project root.

    To verify the integrity of your downloaded files, use the sha256sum command against the prefix included in the filename.

    ln -s YOUR_MODEL_PATH pretrained
# To verify integrity:
sha256sum filename

  10. Evaluate DL3DV models (various view counts)

    main

    Use these commands to evaluate models on the DL3DV dataset with different numbers of input context views (6, 4, or 2). These correspond to Table 7 of the DepthSplat paper.

    # 6 input views:
CUDA_VISIBLE_DEVICES=0 python -m src.main +experiment=dl3dv \
mode=test \
dataset/view_sampler=evaluation \
dataset.view_sampler.num_context_views=6 \
dataset.view_sampler.index_path=assets/dl3dv_start_0_distance_50_ctx_6v_video_0_50.json \
model.encoder.num_scales=2 \
model.encoder.upsample_factor=4 \
model.encoder.lowest_feature_resolution=8 \
model.encoder.monodepth_vit_type=vitb \
checkpointing.pretrained_model=pretrained/depthsplat-gs-base-dl3dv-256x448-randview2-6-02c7b19d.pth

# 4 input views:
CUDA_VISIBLE_DEVICES=0 python -m src.main +experiment=dl3dv \
mode=test \
dataset/view_sampler=evaluation \
dataset.view_sampler.num_context_views=4 \
dataset.view_sampler.index_path=assets/dl3dv_start_0_distance_50_ctx_4v_video_0_50.json \
model.encoder.num_scales=2 \
model.encoder.upsample_factor=4 \
model.encoder.lowest_feature_resolution=8 \
model.encoder.monodepth_vit_type=vitb \
checkpointing.pretrained_model=pretrained/depthsplat-gs-base-dl3dv-256x448-randview2-6-02c7b19d.pth

# 2 input views:
CUDA_VISIBLE_DEVICES=0 python -m src.main +experiment=dl3dv \
mode=test \
dataset/view_sampler=evaluation \
dataset.view_sampler.num_context_views=2 \
dataset.view_sampler.index_path=assets/dl3dv_start_0_distance_50_ctx_2v_video_0_50.json \
model.encoder.num_scales=2 \
model.encoder.upsample_factor=4 \
model.encoder.lowest_feature_resolution=8 \
model.encoder.monodepth_vit_type=vitb \
checkpointing.pretrained_model=pretrained/depthsplat-gs-base-dl3dv-256x448-randview2-6-02c7b19d.pth

  11. Perform Zero-shot Generalization Evaluation

    main

    Evaluate how models trained on RealEstate10K generalize to other datasets like DL3DV or ACID.

    RealEstate10K to DL3DV (Table 8)

    CUDA_VISIBLE_DEVICES=0 python -m src.main +experiment=dl3dv \
mode=test \
dataset/view_sampler=evaluation \
dataset.view_sampler.num_context_views=2 \
dataset.view_sampler.index_path=assets/dl3dv_start_0_distance_10_ctx_2v_tgt_4v.json \
model.encoder.num_scales=2 \
model.encoder.upsample_factor=2 \
model.encoder.lowest_feature_resolution=4 \
model.encoder.monodepth_vit_type=vitl \
checkpointing.pretrained_model=pretrained/depthsplat-gs-large-re10k-256x256-view2-e0f0f27a.pth

    RealEstate10K to ACID (Table 8)

    CUDA_VISIBLE_DEVICES=0 python -m src.main +experiment=re10k \
mode=test \
dataset.roots=[datasets/acid] \
dataset.view_sampler.index_path=assets/evaluation_index_acid.json \
dataset/view_sampler=evaluation \
dataset.view_sampler.num_context_views=2 \
model.encoder.num_scales=2 \
model.encoder.upsample_factor=2 \
model.encoder.lowest_feature_resolution=4 \
model.encoder.monodepth_vit_type=vitl \
checkpointing.pretrained_model=pretrained/depthsplat-gs-large-re10k-256x256-view2-e0f0f27a.pth

  12. Evaluate RealEstate10K models

    main

    Use these commands to evaluate different model scales (Large, Base, Small) on the RealEstate10K dataset. These commands correspond to Table 1 of the DepthSplat paper.

    # To evaluate the large model:
CUDA_VISIBLE_DEVICES=0 python -m src.main +experiment=re10k \
dataset.test_chunk_interval=1 \
model.encoder.num_scales=2 \
model.encoder.upsample_factor=2 \
model.encoder.lowest_feature_resolution=4 \
model.encoder.monodepth_vit_type=vitl \
checkpointing.pretrained_model=pretrained/depthsplat-gs-large-re10k-256x256-view2-e0f0f27a.pth \
mode=test \
dataset/view_sampler=evaluation

# To evaluate the base model:
CUDA_VISIBLE_DEVICES=0 python -m src.main +experiment=re10k \
dataset.test_chunk_interval=1 \
model.encoder.num_scales=2 \
model.encoder.upsample_factor=2 \
model.encoder.lowest_feature_resolution=4 \
model.encoder.monodepth_vit_type=vitb \
checkpointing.pretrained_model=pretrained/depthsplat-gs-base-re10k-256x256-view2-ca7b6795.pth \
mode=test \
dataset/view_sampler=evaluation

# To evaluate the small model:
CUDA_VISIBLE_DEVICES=0 python -m src.main +experiment=re10k \
dataset.test_chunk_interval=1 \
model.encoder.upsample_factor=4 \
model.encoder.lowest_feature_resolution=4 \
checkpointing.pretrained_model=pretrained/depthsplat-gs-small-re10k-256x256-view2-cfeab6b1.pth \
mode=test \
dataset/view_sampler=evaluation