SynergyNet Documentation

FaceBoxes requires a CPU-based Non-Maximum Suppression (NMS) extension. You can build it using the provided build script in the utils directory or by running the convenience shell script.

# Option 1: Using the build script
cd utils
python3 build.py build_ext --inplace

# Option 2: Using the convenience shell script
sh ./build_cpu_nms.sh

Sim3DR is a 3D renderer implemented in C++ and Cython. To build the extension in-place, run the following command from the Sim3DR directory.

python3 setup.py build_ext --inplace

To train the model from scratch:

1. Complete the setup steps (1-4 of the Single Image Inference Demo).
2. Download the training data from the 3DDFA repository (train_aug_120x120.zip) and extract it under the root folder.
3. Execute the training script:

   bash train_script.sh

Training Details:

• Hyperparameters (learning rate, epochs, GPU device) are defined in train_script.sh.
• Default settings require approximately 19GB of VRAM (e.g., on an RTX 3090) and take about 6 hours.
• Scaling: If using a smaller GPU, decrease the batch size and learning rate proportionally.

To use the FaceBoxes functionality, you must manually download the pre-trained model file FaceBoxesProd.pth from the provided Google Drive link and place it in your project directory.

https://drive.google.com/file/d/1tRVwOlu0QtjvADQ2H7vqrRwsWEmaqioI

To run the face detection demonstration, execute the FaceBoxes.py script using Python 3.

python3 FaceBoxes.py

This feature uses a UV-texture GAN to generate meshes from artistic face datasets.

1. Complete setup steps 1-5 of the Single Image Inference Demo.
2. Download the artistic faces data (from AF-Dataset) and the predicted UV maps. Extract them under the root folder.
3. Run the artistic.py script:

• For a whole folder:

  python artistic.py -f art-all --png

• For a single image:

  python artistic.py -f art-all/122.png

Note: Results are significantly better for images that are close to real faces compared to highly abstract samples.

To generate 3D meshes and renderings from real face images using predicted UV maps:

1. Complete setup steps 1-5 of the Single Image Inference Demo.
2. Download the predicted UV maps and real face images for AFLW2000-3D. Extract them under the root folder.
3. Run the uv_texture_realFaces.py script:

• For a whole folder:

  python uv_texture_realFaces.py -f texture_data/real --png

• For a single image:

  python uv_texture_realFaces.py -f texture_data/real/image00002_real_A.png

Results (3D meshes and renderings) are saved in the inference_output directory.

You can integrate SynergyNet into your own projects using a simplified API. First, install the package in editable mode:

pip install -e .

Requirements: The 3dmm_data folder and the pretrained/ directory containing weights must be present in your environment.

Usage: Import SynergyNet from synergy3DMM to get 3D landmarks, face mesh, and face pose in a single call.

Outputs:

• lmk3d: 3D landmarks as [[y, x, z], 68 (points)].
• mesh: 3D face mesh as [[y, x, z], 53215 (points)].
• pose: Face pose consisting of Euler angles [yaw, pitch, roll] and translation [y, x, z].

import cv2
from synergy3DMM import SynergyNet

model = SynergyNet()
I = cv2.imread('<your image path>')

# Returns 3D landmarks, mesh, and pose
lmk3d, mesh, pose = model.get_all_outputs(I)

To evaluate the model performance using the benchmark script, ensure you have completed the setup steps (1-4 of the Single Image Inference Demo). Run the following command, specifying the pretrained model path with the -w flag:

python benchmark.py -w pretrained/best.pth.tar

Results and visualizations for the first 50 examples will be stored in the results/ directory.