DOCSEARCHbyAlgolia
Submit Docs

3DDFA-V3 Documentation

repository·main·Indexed 18 days ago

https://github.com/wang-zidu/3ddfa-v3

A 3D face reconstruction framework that utilizes facial part segmentation for geometric guidance to capture extreme expressions and improve alignment. The framework supports multiple BFM vertex topologies, provides tools for 2D/3D landmark marching, and includes a reconstruction pipeline with support for ResNet50 and MobileNet-V3 backbones.

Tokens
2K
Snippets
5
Records
10
Agent score
64%

What's inside 3DDFA-V3

  1. 3D Mesh Part Masks and Segmentation

    main

    3DDFA-V3 provides 3D mesh part masks aligned with 2D semantic facial segmentation.

    • Topology: The current version is based on the BFM model (35,709 vertices), which is compatible with topologies used by Deep3D, MGCNet, and HRN.
    • Segmentation: You can generate 2D facial segmentation results based on the 3D mesh.
    • Data: To use these masks, you must download them from the project's assets/ documentation.

    To visualize segmentation via CLI:

    • Use --seg_visible for a single subplot showing only visible parts.
    • Use --seg for 8 subplots showing all parts regardless of visibility.

  2. Understand BFM vertex correspondence indices

    main

    The Basel Face Model (BFM) uses four different topological structures with vertex counts of 53,490, 53,215, 38,365, and 35,709.

    To facilitate switching between these topologies, the repository provides correspondence indices in the format indices_A_B.npy (where A and B are vertex counts). While these and the associated .obj files are not strictly required to run the core 3DDFA_V3 algorithm, they are useful for mapping between different BFM versions.

    Available index files:

    • indices_38365_35709.npy
    • indices_53215_35709.npy
    • indices_53215_38365.npy
    • indices_53490_35709.npy

  4. Download 3DDFA-V3 expression data

    main

    The expression datasets used for facial expression synthesis can be downloaded from Hugging Face. The data includes various facial states such as mouth opening, frowning, and eye closing.

    https://huggingface.co/datasets/Zidu-Wang/3DDFA-V3/tree/main/data

  5. Install and Setup 3DDFA-V3

    main

    To use 3DDFA-V3, follow these steps to set up the environment, install dependencies, and configure the renderers.

    1. Clone and Environment Setup

    # Clone the repo
git clone https://github.com/wang-zidu/3DDFA-V3
cd 3DDFA-V3

# Create and activate conda environment
conda create -n TDDFAV3 python=3.8
conda activate TDDFAV3

    2. Install PyTorch

    Depending on your OS and CUDA version, use one of the following:

    Linux/General (CUDA 10.2):

    pip install torch==1.12.1+cu102 torchvision==0.13.1+cu102 torchaudio==0.12.1 --extra-index-url https://download.pytorch.org/whl/cu102

    Conda alternative:

    conda install pytorch==1.12.1 torchvision==0.13.1 torchaudio==0.12.1 cudatoolkit=10.2 -c pytorch

    Windows 10 (Verified for version 1.10):

    pip install torch==1.10.0+cu102 torchvision==0.11.0+cu102 torchaudio==0.10.0 -f https://download.pytorch.org/whl/torch_stable.html

    3. Install Dependencies and Renderers

    # Install project requirements
pip install -r requirements.txt

# Install nvdiffrast (Recommended renderer)
git clone https://github.com/NVlabs/nvdiffrast.git
cd nvdiffrast
pip install .
cd ..

# Install fast CPU renderer (Fallback if nvdiffrast is unavailable)
cd util/cython_renderer/
python setup.py build_ext -i
cd ../..
cd ..
    # Clone the repo:
git clone https://github.com/wang-zidu/3DDFA-V3
cd 3DDFA-V3

conda create -n TDDFAV3 python=3.8
conda activate TDDFAV3

pip install torch==1.12.1+cu102 torchvision==0.13.1+cu102 torchaudio==0.12.1 --extra-index-url https://download.pytorch.org/whl/cu102

pip install -r requirements.txt

git clone https://github.com/NVlabs/nvdiffrast.git
cd nvdiffrast
pip install .
cd ..

cd util/cython_renderer/
python setup.py build_ext -i
cd ../..
cd ..

  6. Run 3D Face Reconstruction Demos

    main

    After preparing assets and pretrained models (refer to the assets/ README), you can run the reconstruction demo using demo.py.

    Example Command

    python demo.py --inputpath examples/ --savepath examples/results --device cuda --iscrop 1 --detector retinaface --ldm68 1 --ldm106 1 --ldm106_2d 1 --ldm134 1 --seg_visible 1 --seg 1 --useTex 1 --extractTex 1 --backbone resnet50

    Output Files

    For each input image, the following files are generated in the --savepath:

    • image_name.png: Visualization results.
    • image_name.npy: Landmarks, segmentation, and other data.
    • image_name_pcaTex.obj: 3D mesh using texture from the BFM model.
    • image_name_extractTex.obj: 3D mesh using texture extracted from the input image.

  7. Load 3D mesh masks and attributes from face_model.npy

    main

    The face_model.npy file contains the face model and various geometric attributes. You can load it using numpy.

    Key attributes available in the loaded dictionary:

    • model['annotation']: Segmentation annotation indices for 8 facial parts.
    • model['annotation_tri']: Triangle faces for 8 facial parts.
    • model['ldm106']: Vertex indices for 106 landmarks.
    • model['ldm134']: Vertex indices for 134 landmarks.
    • model['ldm68']: Vertex indices for 68 landmarks.
    • model['parallel']: Parallel indices for 33 face profile landmarks, used for dynamic 2D and 3D landmark marching.
    import numpy as np
model = np.load("./assets/face_model.npy", allow_pickle=True).item()

  8. Configure demo.py CLI arguments

    main

    The demo.py script accepts several arguments to control the reconstruction process:

    ArgumentDescription
    --inputpathPath to the test data (must be an image folder).
    --savepathPath to the output directory for results (obj, png files).
    --iscropWhether to crop the input image. Set to false only if images are already cropped/resized to (224,224,3).
    --detectorFace detector for cropping. Supports retinaface (recommended) and mtcnn.
    --ldm68, --ldm106, --ldm106_2d, --ldm134Flags to save and show specific landmarks.
    --backboneReconstruction backbone. Supports resnet50 and mbnetv3.
    --seg_visibleSave/show 2D segmentation with a visible mask. Parts that become invisible due to pose changes are not displayed. All 8 parts shown in one subplot.
    --segSave/show 2D segmentation. Even if parts become invisible due to pose, they are still displayed (from 3D estimation) in 8 separate subplots.
    --useTexSave .obj using texture from the BFM model.
    --extractTexSave .obj using texture extracted from the input image (uses median-filtered-weight PCA texture for blending).

  9. Reference the 3DDFA-V3 asset file structure

    main

    The following files are part of the assets directory:

    FileDescription
    face_model.npyFace model and attributes (segmentation, landmarks, etc.)
    net_recon.pth3DDFA_V3 checkpoint
    net_recon_mbnet.pthMobileNet-V3 backbone (optional)
    large_base_net.pthFace detector weights from HRN
    retinaface_resnet50_2020-07-20_old_torch.pthFace detector weights from RetinaFace
    similarity_Lm3D_all.matUsed for face cropping (from Deep3D)
    indices_*.npyOptional BFM vertex correspondence indices
    meanshape-*.objOptional mean shape files (106, 134, 68, parallel, or seg)

  10. Understand the expression data file structure

    main

    The expression data is organized into several compressed ZIP files, each containing specific facial expressions. The available datasets are:

    • mouth_open.zip: Mouth open expressions
    • ugly_mouth.zip: Specific mouth shapes
    • frown.zip: Frowning expressions
    • eye_close.zip: Eyes closed expressions
    • mouth+eye_close.zip: Combined mouth and eye closed expressions
    • nose.zip: Nose-related expressions
    data
├── mouth_open.zip            #  96,152
├── ugly_mouth.zip            #  72,114
├── frown.zip                 # 216,342
├── eye_close.zip             #  72,114
├── mouth+eye_close.zip       #  96,152
└── nose.zip                  #  24,038