DOCSEARCHbyAlgolia
Submit Docs

GaussianAvatars

repository·main·Indexed 21 days ago

https://github.com/shenhanqian/gaussianavatars

A framework for creating photorealistic, rigged 3D head avatars using 3D Gaussian Splatting. It supports training high-fidelity head models, rendering them with realistic motion, and performing novel-view synthesis, self-reenactment, and cross-identity reenactment. The system integrates with the FLAME 2023 and FLAME 2020 models and provides tools for training via train.py, real-time monitoring with a remote viewer, and offline rendering via render.py.

Tokens
4K
Snippets
12
Records
16
Agent score
77%

What's inside GaussianAvatars

  1. Hardware and Software Requirements for GaussianAvatars

    main

    Before installing, ensure your system meets the following specifications:

    Hardware Requirements

    • GPU: CUDA-ready GPU with Compute Capability 7.0 or higher.
    • VRAM: Minimum 11 GB (tested with RTX 2080Ti).

    Software Requirements

    • Conda: Recommended for environment management.
    • C++ Compiler: Required for PyTorch extensions (Visual Studio for Windows, GCC for Linux).
    • CUDA SDK: Must be installed after the C++ compiler and must be compatible with it.
    • FFMPEG: Required for generating result videos.

    Additional Python Packages

    • RoMa: For rotation representations.
    • DearPyGUI: For the viewer interface.
    • NVDiffRast: For mesh rendering in the viewer.

    Tested Compatibility Matrix

    PyTorch VersionCUDA versionLinuxWindows (VS2022)Windows (VS2019)
    2.0.111.7.1PassFail to compilePass
    2.2.012.1.1PassPassPass

  2. Understand the Gaussian-Splatting License terms

    main

    The gaussian-splatting software is licensed under a specific research-oriented license held by Inria and the Max Planck Institut für Informatik (MPII).

    Permitted Use

    • Research Purposes Only: The software is granted for non-exclusive, royalty-free use by both academic and industrial research users for research and/or evaluation purposes.
    • Derivative Works: You are granted rights to reproduce, prepare derivative works of, publicly display, publicly perform, and distribute your Work (the software and its derivatives) in any form, subject to the limitations below.

    Key Limitations

    • Non-Commercial Restriction: You cannot use, exploit, or distribute the software for commercial purposes without prior and explicit consent from the licensors. For unauthorized commercial use, you must contact Inria at stip-sophia.transfert@inria.fr.
    • Redistribution Requirements: If you redistribute the Work, you must:
      1. Do so under this same License.
      2. Include a complete copy of this License.
      3. Retain all original copyright, patent, trademark, or attribution notices.
    • Derivative Terms: You may apply different terms to your derivative works only if those terms maintain the same use limitations (research/evaluation only) and you clearly identify which works are subject to your terms.

    Citation Requirement

    When using the software for publications or presenting results obtained through its use, users are strongly encouraged to cite the corresponding publications as specified in the software's documentation.

  3. Obtain Human Head Video Data

    main

    To use the pre-processed datasets or custom data for training/rendering, you can acquire data through the following methods:

    1. Preprocessed NeRSemble Dataset

    The project uses 9 subjects from the NeRSemble dataset. You can download the pre-processed versions from:

    Note: If using this pre-processed data, please cite the NeRSemble dataset in your work.

    2. Custom Data

    To use your own video data, you must first preprocess it using the VHAP head-tracking pipeline.

  4. Install GaussianAvatars via Conda

    main

    Follow these steps to set up the gaussian-avatars environment. This process involves cloning the repository, creating a Conda environment, installing the CUDA toolkit and ninja for compilation, and configuring environment variables for your specific OS.

    1. Clone and Create Environment

    git clone https://github.com/ShenhanQian/GaussianAvatars.git --recursive
cd GaussianAvatars

conda create --name gaussian-avatars -y python=3.10
conda activate gaussian-avatars

# Install CUDA and ninja for compilation (ensure version matches your needs)
conda install -c "nvidia/label/cuda-11.7.1" cuda-toolkit ninja

    2. Configure Environment Variables

    Depending on your operating system, you must set specific paths for CUDA and your C++ compiler to ensure PyTorch extensions compile correctly.

    Linux

    ln -s "$CONDA_PREFIX/lib" "$CONDA_PREFIX/lib64"  # Avoids '/usr/bin/ld: cannot find -lcudart'
conda env config vars set CUDA_HOME=$CONDA_PREFIX

    Windows (PowerShell)

    conda env config vars set CUDA_PATH="$env:CONDA_PREFIX"  

# For Visual Studio 2022 (Update the version number 14.39.33519 to match your installation)
conda env config vars set PATH="$env:CONDA_PREFIX\Script;C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\bin\Hostx64\x64;$env:PATH"

# OR For Visual Studio 2019 (Update the version number 14.29.30133 to match your installation)
conda env config vars set PATH="$env:CONDA_PREFIX\Script;C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\bin\HostX86\x86;$env:PATH" 

# Re-activate to apply changes
conda deactivate
conda activate gaussian-avatars

    Windows (Command Prompt)

    conda env config vars set CUDA_PATH=%CONDA_PREFIX%

# For Visual Studio 2022 (Update the version number 14.39.33519 to match your installation)
conda env config vars set PATH="%CONDA_PREFIX%\Script;C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\bin\Hostx64\x64;%PATH%"

# OR For Visual Studio 2019 (Update the version number 14.29.30133 to match your installation)
conda env config vars set PATH="%CONDA_PREFIX%\Script;C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\bin\HostX86\x86;%PATH%"

# Re-activate to apply changes
conda deactivate
conda activate gaussian-avatars

    3. Install PyTorch and Dependencies

    Ensure the PyTorch CUDA version matches the toolkit installed in Step 1.

    # Install PyTorch via pip
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu117

# OR Install PyTorch via conda
conda install pytorch torchvision pytorch-cuda=11.7 -c pytorch -c nvidia

# Verify CUDA availability
# python -c "import torch; print(torch.cuda.is_available())"

# Install remaining requirements (this will compile diff-gaussian-rasterization, simple-knn, and nvdiffrast)
pip install -r requirements.txt
    git clone https://github.com/ShenhanQian/GaussianAvatars.git --recursive
cd GaussianAvatars

conda create --name gaussian-avatars -y python=3.10
conda activate gaussian-avatars

conda install -c "nvidia/label/cuda-11.7.1" cuda-toolkit ninja

  5. Train a GaussianAvatar

    main

    Use train.py to train the 3D Gaussian model. You can specify the source data, model output directory, and various training parameters such as binding to a mesh or using a white background.

    Example Command:

    SUBJECT=306

python train.py \
-s data/UNION10_${SUBJECT}_EMO1234EXP234589_v16_DS2-0.5x_lmkSTAR_teethV3_SMOOTH_offsetS_whiteBg_maskBelowLine \
-m output/UNION10EMOEXP_${SUBJECT}_eval_600k \
--eval --bind_to_mesh --white_background --port 60000

  6. Perform offline rendering with render.py

    main

    Use render.py to generate renderings from a trained model. You can perform novel-view synthesis, self-reenactment, or cross-identity reenactment by specifying the model path and various flags to include or skip specific datasets (train, val, test).

    python render.py -m <path to trained model>

  7. Use the Remote Viewer to monitor training

    main

    The remote viewer allows you to monitor training progress in real-time via a GUI.

    Note: The remote viewer can significantly slow down training. It is recommended to close it or use the "pause rendering" option when not actively viewing. If the viewer freezes when enabling "show mesh" for the first time, try toggling the option or waiting a few seconds.

    python remote_viewer.py --port 60000

  8. Use the Local Viewer to render trained Gaussians

    main

    After training is complete, use local_viewer.py to load and render the optimized 3D Gaussians.

    Example Command:

    SUBJECT=306
ITER=300000

python local_viewer.py \
--point_path output/UNION10EMOEXP_${SUBJECT}_eval_600k/point_cloud/iteration_${ITER}/point_cloud.ply

    Warning: The viewer is implemented in Python and is not suitable for performance benchmarking. For accurate FPS evaluation, use the Offline Rendering FPS benchmark script.

    Arguments:

    • --point_path: Path to the gaussian splatting file (.ply).
    • --motion_path: Path to a motion file (.npz). Required only if loading a different motion sequence than the one used during training.

  9. Download and configure the FLAME model

    main

    The GaussianAvatars code and pre-processed data require the FLAME 2023 model. You must download the original assets from the FLAME website and place them in the following directory structure:

    • FLAME 2023 (with jaw rotation): flame_model/assets/flame/flame2023.pkl
    • FLAME Vertex Masks: flame_model/assets/flame/FLAME_masks.pkl

    Using FLAME 2020: If you intend to use FLAME 2020 instead, follow these steps:

    1. Download the FLAME 2020 model to flame_model/assets/flame/generic_model.pkl.
    2. Update the FLAME_MODEL_PATH environment variable or configuration in flame_model/flame.py.
    3. Ensure your FLAME tracking results are also based on FLAME 2020.
    flame_model/assets/flame/flame2023.pkl
flame_model/assets/flame/FLAME_masks.pkl

  12. Perform Self-Reenactment

    main

    To render the test set for self-reenactment, use the --skip_train and --skip_val flags. You can also restrict rendering to a specific camera view (e.g., front view) using --select_camera_id.

    SUBJECT=306

# Render the test set
python render.py \
-m output/UNION10EMOEXP_${SUBJECT}_eval_600k \
--skip_train --skip_val

# Render the test set only in a front view (camera id 8)
python render.py \
-m output/UNION10EMOEXP_${SUBJECT}_eval_600k \
--skip_train --skip_val \
--select_camera_id 8