DOCSEARCHbyAlgolia
Submit Docs

SadTalker: Stylized Audio-Driven Talking Face Animation

repository·main·Indexed 12 days ago

https://github.com/opentalker/sadtalker

SadTalker generates realistic talking head videos from a single portrait image and an audio file. It features a Gradio WebUI and a CLI for portrait animation, supporting face enhancement via GFPGAN and RestoreFormer, background enhancement via Real-ESRGAN, and 4D free-view talking head generation. The project also includes Arcface Torch for distributed face recognition training and 3D face visualization capabilities.

Tokens
16.2K
Snippets
59
Records
77
Agent score
95%

What's inside SadTalker

  1. What is Partial FC and when to use it

    main

    Partial FC is a sparse variant of the model parallel architecture designed for large-scale face recognition. It uses a sparse softmax where each batch dynamically samples a subset of class centers for training.

    Key Benefits:

    • Scalability: Allows training on datasets with massive identity counts (up to 29 million identities).
    • Efficiency: Reduces GPU memory consumption and computational overhead compared to standard Data Parallel or Model Parallel methods.
    • Performance: Provides significantly faster training speeds and lower memory costs when the number of identities in the dataset exceeds 300K.

    Comparison Summary (Tesla V100 32GB * 8):

    • Data Parallel: Limited by GPU memory as identity counts increase; fails for large datasets.
    • Model Parallel: Higher memory cost than Partial FC; also fails for very large datasets.
    • Partial FC 0.1: Maintains high training speed and manageable memory usage even as identities scale to tens of millions.

  2. Choose the correct `--preprocess` mode

    main

    The --preprocess flag determines how the input image is processed before animation. Choosing the right mode depends on your source image type:

    • crop (Default): Generates animation based on the facial keypoints of the cropped area. This produces realistic expression and head pose animation. Note: Using --still in this mode will stop eyeblink and head pose movement.
    • resize: Resizes the entire image to a specific resolution (similar to an ID photo). Warning: This produces poor results for full-body or full-person images; it is best suited for portrait/ID-style photos.
    • full: Processes the cropped region and pastes it back onto the original image. For best results with full-body images, use this in combination with --still to maintain the original head pose.

  3. Compare Model Parallel vs Partial FC 0.1 performance

    main

    Based on benchmarks using 8x Tesla V100 32GB GPUs, Partial FC 0.1 provides significant advantages over Model Parallel training, especially as the number of identities in the dataset increases:

    • Speed: Partial FC 0.1 is significantly faster. For example, at 3 million identities, Partial FC 0.1 achieves ~5300 samples/sec compared to ~2270 samples/sec for Model Parallel.
    • Memory Efficiency: Partial FC 0.1 uses substantially less GPU memory. At 3 million identities, it uses approximately 1/3 of the memory required by Model Parallel.
    • Scalability: Partial FC 0.1 scales to much larger datasets (up to 29 million identities) where Model Parallel and Data Parallel methods fail due to memory constraints.

  4. Install SadTalker on Windows WSL

    main

    When using Windows Subsystem for Linux (WSL), you must ensure the following environment variable is set to allow proper library loading:

    export LD_LIBRARY_PATH=/usr/lib/wsl/lib:$LD_LIBRARY_PATH

    export LD_LIBRARY_PATH=/usr/lib/wsl/lib:$LD_LIBRARY_PATH

  5. Install arcface_torch for v1.8.0

    main

    To install arcface_torch for version 1.8.0 on Linux or Windows, use the following commands based on your CUDA version or if you are using a CPU-only environment. Note that a default timeout of 100 seconds is used to ensure stability during download.

    # CUDA 11.0
pip --default-timeout=100 install torch==1.8.0+cu111 torchvision==0.9.0+cu111 torchaudio==0.8.0 -f https://download.pytorch.org/whl/torch_stable.html

# CUDA 10.2
pip --default-timeout=100 install torch==1.8.0 torchvision==0.9.0 torchaudio==0.8.0

# CPU only
pip --default-timeout=100 install torch==1.8.0+cpu torchvision==0.9.0+cpu torchaudio==0.8.0 -f https://download.pytorch.org/whl/torch_stable.html

  6. Enhance face and background resolution

    main

    To achieve higher resolution outputs, you can integrate face and background restoration networks.

    1. Face Enhancement: Use --enhancer <gfpgan or RestoreFormer> to enhance the generated face.
    2. Background Enhancement: Use --background_enhancer <realesrgan> to enhance the entire video frame.

    Ensure the required packages are installed via pip:

    # make sure above packages are available:
pip install gfpgan
pip install realesrgan

  7. Download SadTalker Checkpoints

    main

    Before running the extension, you must download the model checkpoints.

    Linux and macOS: Run the following command in your terminal to download the models using the provided script:

    cd SOMEWHERE_YOU_LIKE
bash <(wget -qO- https://raw.githubusercontent.com/Winfredy/OpenTalker/main/scripts/download_models.sh)

    Windows: Download all checkpoints manually from the SadTalker model download section.

    bash
cd SOMEWHERE_YOU_LIKE
bash <(wget -qO- https://raw.githubusercontent.com/Winfredy/OpenTalker/main/scripts/download_models.sh)

  8. Run SadTalker using Docker

    main

    You can use a community Docker image (wawa9000/sadtalker) to run SadTalker without local installation. Use the --gpus "all" flag to enable GPU acceleration and mount your local directory to /host_dir to access input files and save results.

    docker run --gpus "all" --rm -v $(pwd):/host_dir wawa9000/sadtalker \
    --driven_audio /host_dir/deyu.wav \
    --source_image /host_dir/image.jpg \
    --expression_scale 1.0 \
    --still \
    --result_dir /host_dir

  9. Download SadTalker models

    main

    You can automatically download all required models on Linux or macOS using the provided script. For manual downloads, checkpoints are available via Google Drive, GitHub Releases, or Baidu Netdisk.

    Additionally, an offline patch for gfpgan/ is provided to avoid downloading models during generation if you prefer an offline setup.

    bash scripts/download_models.sh

  10. Test training performance for ArcFace models

    main

    To benchmark training performance, you can compare Model Parallel training against Partial FC 0.1 training.

    Benchmark configuration used in examples:

    • Identities: 3 million (synthetic data)
    • Precision: Mixed precision enabled
    • Backbone: ResNet50
    • Batch Size: 1024

    Use torch.distributed.launch to run the training scripts across multiple GPUs.

    # Model Parallel training
python -m torch.distributed.launch --nproc_per_node=8 --nnodes=1 --node_rank=0 --master_addr="127.0.0.1" --master_port=1234 train.py configs/3millions

# Partial FC 0.1 training
python -m torch.distributed.launch --nproc_per_node=8 --nnodes=1 --node_rank=0 --master_addr="127.0.0.1" --master_port=1234 train.py configs/3millions_pfc

  11. Install requirements for 3D Face Visualization

    main

    To enable 3D face visualization using pytorch3d, you must set up a specific Conda environment and install pre-compiled wheels. Follow these steps to ensure compatibility with the required PyTorch and CUDA versions:

    1. Create and activate a Python 3.8 environment.
    2. Install system dependencies (ffmpeg, fvcore, iopath, libgcc, gmp) via Conda.
    3. Install the specific PyTorch version (1.11.0+cu113).
    4. Install pytorch3d using the provided pre-compiled wheel link.
    5. Install additional requirements from requirements3d.txt.
    git clone https://github.com/OpenTalker/SadTalker.git
cd SadTalker 
conda create -n sadtalker3d python=3.8
source activate sadtalker3d

conda install ffmpeg
conda install -c fvcore -c iopath -c conda-forge fvcore iopath
conda install libgcc gmp

pip install torch==1.11.0+cu113 torchvision==0.12.0+cu113 torchaudio==0.11.0 --extra-index-url https://download.pytorch.org/whl/cu113

# install pytorch3d
pip install --no-index --no-cache-dir pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/py38_cu113_pyt1110/download.html

pip install -r requirements3d.txt

  12. Install arcface_torch for v1.7.1

    main

    To install arcface_torch for version 1.7.1 on Linux or Windows, select the command corresponding to your CUDA version (11.0, 10.2, 10.1, or 9.2) or use the CPU-only installation.

    # CUDA 11.0
pip install torch==1.7.1+cu110 torchvision==0.8.2+cu110 torchaudio==0.7.2 -f https://download.pytorch.org/whl/torch_stable.html

# CUDA 10.2
pip install torch==1.7.1 torchvision==0.8.2 torchaudio==0.7.2

# CUDA 10.1
pip install torch==1.7.1+cu101 torchvision==0.8.2+cu101 torchaudio==0.7.2 -f https://download.pytorch.org/whl/torch_stable.html

# CUDA 9.2
pip install torch==1.7.1+cu92 torchvision==0.8.2+cu92 torchaudio==0.7.2 -f https://download.pytorch.org/whl/torch_stable.html

# CPU only
pip install torch==1.7.1+cpu torchvision==0.8.2+cpu torchaudio==0.7.2 -f https://download.pytorch.org/whl/torch_stable.html