DOCSEARCHbyAlgolia
Submit Docs

photometric_optimization

repository·master·Indexed 20 days ago

https://github.com/havenfeng/photometric_optimization

An analysis-by-synthesis framework for fitting a textured FLAME 3D head model to images. It includes methods for building a custom FLAME texture space from in-the-wild images and optimizing for FLAME parameters, appearance, and lighting.

Tokens
579
Snippets
2
Records
4
Agent score
20%

What's inside photometric_optimization

  1. How the FLAME texture space is built

    master

    The project describes a pipeline to build a FLAME texture space from in-the-wild images (specifically using the FFHQ dataset) to cover diverse ethnicities and age groups. The process follows four main stages:

    1. Initialization: An initial texture basis is created by fitting FLAME to the Basel Face Model (BFM) template and projecting BFM vertex colors onto the FLAME mesh.
    2. Model Fitting: FLAME is fitted to FFHQ images by optimizing shape, pose, expression, initial texture space parameters, Spherical Harmonics (SH) lighting (9 coefficients), and a texture offset. The loss function combines landmark loss (using FAN), photometric loss (restricted to skin regions via face segmentation), and various regularizers.
    3. Texture Completion: To handle occlusions, an inpainting network (adapted from GMCNN) is trained to fill in missing regions of the texture maps using random strokes in the visible face regions.
    4. Texture Space Computation: Principal Component Analysis (PCA) is applied to the completed texture maps to compute the final texture space.

  3. Setup the single image photometric fitting demo

    master

    To run the photometric fitting demo, you need a Conda environment with specific versions of PyTorch and PyTorch3D. It is recommended to install PyTorch3D via Conda for better CUDA support. The environment requires Python 3.8, PyTorch 1.5.0, and PyTorch3D 0.2.

    conda create -n pytorch3d python=3.8
conda activate pytorch3d
conda install -c pytorch pytorch=1.5.0 torchvision cudatoolkit=10.2
conda install -c conda-forge -c fvcore fvcore
conda install pytorch3d -c pytorch3d