DOCSEARCHbyAlgolia
Submit Docs

3D Gaussian Splatting

repository·main·Indexed 12 days ago

https://github.com/graphdeco-inria/gaussian-splatting

A real-time radiance field rendering method for high-quality novel-view synthesis at 1080p resolution (≥ 30 fps). The system includes a PyTorch-based optimizer for generating 3D Gaussian models from Structure-from-Motion (SfM) inputs, an OpenGL-based real-time viewer, and tools for dataset conversion and evaluation via train.py, render.py, and metrics.py.

Tokens
6.5K
Snippets
19
Records
28
Agent score
48%

What's inside 3D Gaussian Splatting

  1. Overview of Gaussian Splatting components

    main

    The codebase is organized into four primary functional components:

    1. PyTorch-based optimizer: Generates a 3D Gaussian model starting from Structure-from-Motion (SfM) inputs.
    2. Network viewer: Allows you to connect to and visualize the optimization process as it happens.
    3. OpenGL-based real-time viewer: Used to render trained models in real-time.
    4. Dataset script: A utility to convert your own images into SfM datasets ready for optimization.

    Note that these components have varying hardware and software requirements. They have been tested on Windows 10 and Ubuntu Linux 22.04.

  2. Install the Gaussian Splatting Environment via Conda

    main

    The recommended installation method uses Conda to manage packages and environments.

    Note for Windows users: You must set the DISTUTILS_USE_SDK environment variable before creating the environment.

    Note on CUDA version: This process assumes CUDA SDK 11 is installed. It is not compatible with CUDA 12 via this specific method.

    To save disk space, you can specify a different package download location and environment prefix on a different drive.

    # Windows only
SET DISTUTILS_USE_SDK=1
conda env create --file environment.yml
conda activate gaussian_splatting

# To use a different drive for packages and environment:
conda config --add pkgs_dirs <Drive>/<pkg_path>
conda env create --file environment.yml --prefix <Drive>/<env_path>/gaussian_splatting
conda activate <Drive>/<env_path>/gaussian_splatting

  3. Clone the Gaussian Splatting repository

    main

    The repository contains submodules, so you must use the --recursive flag when cloning to ensure all necessary components are downloaded. You can clone via SSH or HTTPS.

    # SSH
git clone git@github.com:graphdeco-inria/gaussian-splatting.git --recursive

# HTTPS
git clone https://github.com/graphdeco-inria/gaussian-splatting --recursive

  4. Evaluate Trained Models with render.py and metrics.py

    main

    To evaluate models using a train/test split (enabled via --eval during training), follow this workflow:

    1. Train: python train.py -s <path> --eval
    2. Render: Generate renderings using render.py.
    3. Metrics: Compute error metrics using metrics.py.

    Evaluating Pre-trained Models: When using pre-trained models, you must provide the original source data using the -s flag to render.py.

    Note: Metrics obtained from this codebase may differ from the original paper due to bugfixes and code cleanup.

    # 1. Train with a split
python train.py -s <path to COLMAP or NeRF Synthetic dataset> --eval

# 2. Generate renderings
python render.py -m <path to trained model>

# 3. Compute error metrics
python metrics.py -m <path to trained model>

# --- Evaluating Pre-trained Models ---

# Provide source data to render.py
python render.py -m <path to pre-trained model> -s <path to COLMAP dataset>
python metrics.py -m <path to pre-trained model>

  5. Prepare your own scenes using convert.py

    main

    To use custom images for training, you must convert them into a COLMAP-compatible dataset structure. The loaders expect a specific hierarchy containing images and sparse/0 (with cameras.bin, images.bin, and points3D.bin).

    Standard Workflow

    1. Place your raw images in <location>/input.
    2. Ensure COLMAP and ImageMagick are installed on your system path.
    3. Run the conversion script:
    python convert.py -s <location> [--resize]

    Note: Use --resize to create multi-resolution versions (1/2, 1/4, 1/8) for faster/better training. This requires ImageMagick.

    Using existing COLMAP data (without undistortion)

    If you already have COLMAP data (e.g., using OPENCV camera models) and want to skip the matching step:

    1. Place images in <location>/input.
    2. Place COLMAP info in <location>/distorted (containing database.db and sparse/0).
    3. Run:
    python convert.py -s <location> --skip_matching [--resize]
    python convert.py -s <location> [--resize]

  7. Use the SIBR Real-Time Viewer

    main

    The Real-Time Viewer is used to visualize trained models.

    To run the app, provide the path to your trained model using the -m flag:

    ./<SIBR install dir>/bin/SIBR_gaussianViewer_app -m <path to trained model>

    Performance Tips

    • V-Sync: Disable V-Sync on your machine and in the application (Menu $\rightarrow$ Display) to unlock full frame rate.
    • Multi-GPU: In multi-GPU systems (like laptops), ensure your OpenGL/Display GPU is the same as your CUDA GPU for maximum performance.
    • Culling: The viewer uses aggressive fast culling. If you encounter issues, you can toggle this in the floating menu.

    Navigation

    • FPS Navigator (Default):
      • Translation: W, A, S, D, Q, E
      • Rotation: I, K, J, L, U, O
    • Trackball Navigator: Select from the floating menu.
    • Camera Snapping: Use Snap to or Snap to closest buttons.
    • Scaling: Use the Scaling Modifier to control Gaussian size.
    # Example: Running the real-time viewer with a trained model
./<SIBR install dir>/bin/SIBR_gaussianViewer_app -m /path/to/model_dir

  8. Enable anti-aliasing with EWA Filter

    main

    To remove aliasing artifacts, you can enable the EWA Filter (from Mip Splatting). This is disabled by default.

    Usage

    Add the --antialiasing flag to either train.py or render.py:

    python train.py --antialiasing ...

    Note: If you train with anti-aliasing, you must also enable it in the SIBR viewer to see the effect correctly.

    python train.py --antialiasing

  9. Enable depth regularization for better reconstruction

    main

    Depth regularization uses depth maps as priors to improve reconstruction, especially in untextured areas like roads, and can help remove 'floaters'.

    Setup for Real-World Datasets

    1. Clone Depth Anything v2: git clone https://github.com/DepthAnything/Depth-Anything-V2.git
    2. Download Weights: Place depth_anything_v2_vitl.pth in Depth-Anything-V2/checkpoints/.
    3. Generate Depth Maps:
      python Depth-Anything-V2/run.py --encoder vitl --pred-only --grayscale --img-path <path to input images> --outdir <output path>
    4. Generate Scale Parameters:
      python utils/make_depth_scale.py --base_dir <path to colmap> --depths_dir <path to generated depths>

    Training

    Pass the path to the generated depth maps using the -d flag:

    --d <path to depth maps>
    python Depth-Anything-V2/run.py --encoder vitl --pred-only --grayscale --img-path <path to input images> --outdir <output path>

  10. Optimize training for large-scale datasets

    main

    When training on large-scale datasets (e.g., city districts) with multi-scale details like extreme close-ups mixed with far-away shots, the default learning rates may cause issues. To improve results in these scenarios, you should lower the learning rates for position and scaling.

    Recommended adjustments:

    • Decrease --position_lr_init (e.g., to 0.000016)
    • Decrease --position_lr_final (e.g., to 0.0000016)
    • Decrease --scaling_lr (e.g., to 0.001)

    The more extensive the scene, the lower these values should be.

    # Example of reduced learning rates for large scenes
python train.py --position_lr_init 0.000016 --scaling_lr 0.001

  11. Hardware and Software Requirements for Training

    main

    To run the 3D Gaussian Splatting optimizer, your system must meet the following requirements:

    Hardware

    • GPU: CUDA-ready GPU with Compute Capability 7.0 or higher.
    • VRAM: 24 GB recommended to achieve paper-quality results (refer to FAQ for lower VRAM configurations).

    Software

    • Environment Manager: Conda (recommended).
    • C++ Compiler: Required for PyTorch extensions (Visual Studio 2019 is used for Windows).
    • CUDA SDK: Version 11 is required (specifically 11.8; version 11.6 is known to have issues). The CUDA SDK must be installed after Visual Studio and must be compatible with the C++ compiler.

  12. Install SIBR Interactive Viewers

    main

    The project provides two interactive viewers based on the SIBR framework: remote and real-time.

    Hardware Requirements

    • GPU: OpenGL 4.5-ready GPU and drivers (or latest MESA software).
    • VRAM: 4 GB recommended.
    • Real-Time Viewer only: CUDA-ready GPU with Compute Capability 7.0+.

    Software Requirements

    • Compiler: Visual Studio or g++ (not Clang). Windows users should use Visual Studio 2019.
    • CUDA SDK: Version 11 (install after Visual Studio; 11.8 used in testing).
    • CMake: Recent version (3.24 used in testing).
    • Windows only: 7zip.

    Installation Methods

    1. Windows (Recommended)

    Use pre-built binaries to avoid complex dependency compilation: Download Viewers.

    2. Build from Source (Windows)

    If you cloned with --recursive, the source is in SIBR_viewers:

    cd SIBR_viewers
cmake -Bbuild .
cmake --build build --target install --config RelWithDebInfo

    Note: You can use Debug instead of RelWithDebInfo for development.

    3. Build from Source (Ubuntu 22.04)

    Install dependencies first:

    sudo apt install -y libglew-dev libassimp-dev libboost-all-dev libgtk-3-dev libopencv-dev libglfw3-dev libavdevice-dev libavcodec-dev libeigen3-dev libxxf86vm-dev libembree-dev

    Then build:

    cd SIBR_viewers
cmake -Bbuild . -DCMAKE_BUILD_TYPE=Release
cmake --build build -j24 --target install

    Tip: Add -G Ninja to the cmake command to build faster.

    4. Build from Source (Ubuntu 20.04)

    For compatibility with Focal Fossa, check out the compatibility branch before building:

    git checkout fossa_compatibility