SyncTalk Documentation

To use the DeepSpeech feature processing routines, install the necessary dependencies using pip from the requirements.txt file.

pip3 install -r requirements.txt

To prepare a video for training or inference, use data_utils/process.py.

Requirements:

• Video must be 25FPS.
• All frames must contain the talking person.
• Resolution should be approximately 512x512.
• Duration should be about 4-5 minutes.

Command:

python data_utils/process.py data/<ID>/<ID>.mp4 --asr ave

ASR Options (--asr):

• ave: Audio Visual Encoder (recommended for accurate lip sync).
• deepspeech
• hubert

Processed videos are saved in the data folder.

Before running SyncTalk, you must organize your data and models as follows:

1. Pre-trained Models:

   • Place May.zip in the data folder and unzip it.
   • Place trial_may.zip in the model folder and unzip it.

2. Face Parsing Model:

   wget https://github.com/YudongGuo/AD-NeRF/blob/master/data_util/face_parsing/79999_iter.pth?raw=true -O data_utils/face_parsing/79999_iter.pth

3. 3DMM Model for Head Pose Estimation: Download the following files to data_utils/face_tracking/3DMM/:

   • exp_info.npy
   • keys_info.npy
   • sub_mesh.obj
   • topology_info.npy

4. Basel Face Model (BFM):

   • Copy 01_MorphableModel.mat to data_util/face_tracking/3DMM/.
   • Run the conversion script:

     cd data_utils/face_tracking
     python convert_BFM.py

For Windows users, a pre-integrated package is available. Download SyncTalk-Windows.zip, unzip the contents, and run the inference.bat file to start the demo.

Download Links:

• Hugging Face
• Baidu Netdisk

double-click `inference.bat` to run the demo

If the head-only trained model produces a "double chin" effect, you can perform torso training.

Warning: When using torso training, you cannot use the --portrait mode. Adding --portrait will cause the torso model to fail.

Torso Training

Pass the latest head checkpoint using --head_ckpt:

# Example: training torso using a head checkpoint
python main.py data/May/ --workspace model/trial_may_torso/ -O --torso --head_ckpt model/trial_may/ngp_ep0019.pth --iters 150000 --asr_model ave

Torso Inference/Testing

# Test torso model
python main.py data/May/ --workspace model/trial_may_torso/ -O --torso --test --asr_model ave

# Inference with target audio
python main.py data/May/ --workspace model/trial_may_torso/ -O --torso --test --test_train --asr_model ave --aud ./demo/test.wav

The main.py script serves as the primary entrypoint for the SyncTalk pipeline, supporting training, testing, and GUI modes. The script requires a path argument (likely the dataset path) and provides extensive configuration via command-line flags.

Common Execution Modes

• Training: The default mode when no specific test flags are provided.
• Testing: Use --test to load a model and run on the test dataset. Use --test_train to load a model and test specifically on the training dataset.
• GUI Mode: Use --gui to launch an interactive GUI for rendering and inspection.

Important Hardware Note

To prevent low numerical accuracy on RTX 30xx series GPUs, the script automatically attempts to disable tf32 features:

torch.backends.cuda.matmul.allow_tf32 = False
torch.backends.cudnn.allow_tf32 = False

Evaluation

Run evaluation on a processed subject:

# Standard evaluation
python main.py data/May --workspace model/trial_may -O --test --asr_model ave

# Evaluation with Portrait mode (pasting face back onto original image for higher quality)
python main.py data/May --workspace model/trial_may -O --test --asr_model ave --portrait

Inference with Target Audio

To generate a talking head from a specific audio file:

python main.py data/May --workspace model/trial_may -O --test --test_train --asr_model ave --portrait --aud ./demo/test.wav

• Use .wav files for inference.
• Results are saved in model/trial_may/results/.
• If not using the Audio Visual Encoder (ave), replace the .wav path with the corresponding .npy file path.

Use extract_ds_features.py to generate DeepSpeech features from a directory of input files. Provide the path to your data directory using the --input flag.

python3 extract_ds_features.py --input=<you_data_dir>

Use extract_wav.py to generate .wav files from a directory of video data. Provide the path to your data directory using the --in-video flag.

python3 extract_wav.py --in-video=<you_data_dir>

The infer_bs(root_path) function is the core logic for the blendshape capture process. It initializes a MediaPipe FaceLandmarker with output_face_blendshapes=True and output_facial_transformation_matrixes=True.

It iterates through all .mp4 files in the root_path. For each video, it:

• Checks if bs.npy already exists to avoid redundant processing.
• Extracts blendshape scores from the MediaPipe result.
• Slices the scores (skipping the first index) and appends a zero to maintain a specific coefficient shape.
• Smooths the coefficients using scipy.signal.savgol_filter.
• Saves the final array to bs.npy using np.save.