DOCSEARCHbyAlgolia
Submit Docs

VideoReTalking

repository·main·Indexed 27 days ago

https://github.com/opentalker/video-retalking

A system for audio-based lip synchronization and talking head video editing. It enables editing faces in real-world videos to match input audio, supporting facial expression modifications via CLI arguments and upper face adjustments using GANimation. The repository includes components for face detection, Arcface Torch model training with Partial FC for large-scale datasets, and integration with GFPGAN.

Tokens
5.2K
Snippets
17
Records
25
Agent score
92%

What's inside VideoReTalking

  2. Understand Partial FC for large-scale training

    main

    When training with more than 300K identities, the Partial FC strategy is recommended. It is a sparse variant of model parallel architecture that uses a sparse softmax to dynamically sample a subset of class centers. This results in:

    • Faster training performance.
    • Significantly lower GPU memory consumption.
    • Ability to scale to datasets with up to 29 million identities.
    • Support for multi-machine distributed training and mixed precision training.

  3. Install and Setup Face Morpher

    main

    To use Face Morpher, you need to install the dependencies and download the required dlib shape predictor file.

    1. Install requirements:
    pip install -r requirements.txt
    1. Download the shape predictor file:

    http://dlib.net/files/shape_predictor_68_face_landmarks.dat.bz2 and extract it. 3. Set the DLIB_DATA_DIR environment variable to the directory containing shape_predictor_68_face_landmarks.dat. For example:

    export DLIB_DATA_DIR=/path/to/your/data

  8. Install PyTorch v1.6.0 for arcface_torch

    main

    Use the following commands to install PyTorch v1.6.0 on Linux or Windows, depending on your CUDA version or if you are using a CPU only.

    # CUDA 10.2
pip install torch==1.6.0 torchvision==0.7.0

# CUDA 10.1
pip install torch==1.6.0+cu101 torchvision==0.7.0+cu101 -f https://download.pytorch.org/whl/torch_stable.html

# CUDA 9.2
pip install torch==1.6.0+cu92 torchvision==0.7.0+cu92 -f https://download.pytorch.org/whl/torch_stable.html

# CPU only
pip install torch==1.6.0+cpu torchvision==0.7.0+cpu -f https://download.pytorch.org/whl/torch_stable.html

  9. Install PyTorch v1.7.1 for arcface_torch

    main

    Use the following commands to install PyTorch v1.7.1 on Linux or Windows, depending on your CUDA version or if you are using a CPU only.

    # CUDA 11.0
pip install torch==1.7.1+cu110 torchvision==0.8.2+cu110 torchaudio==0.7.2 -f https://download.pytorch.org/whl/torch_stable.html

# CUDA 10.2
pip install torch==1.7.1 torchvision==0.8.2 torchaudio==0.7.2

# CUDA 10.1
pip install torch==1.7.1+cu101 torchvision==0.8.2+cu101 torchaudio==0.7.2 -f https://download.pytorch.org/whl/torch_stable.html

# CUDA 9.2
pip install torch==1.7.1+cu92 torchvision==0.8.2+cu92 torchaudio==0.7.2 -f https://download.pytorch.org/whl/torch_stable.html

# CPU only
pip install torch==1.7.1+cpu torchvision==0.8.2+cpu torchaudio==0.7.2 -f https://download.pytorch.org/whl/torch_stable.html

  11. Train Arcface Torch models

    main

    Training is performed by running train.py and providing a path to a configuration file. The library supports distributed training across single or multiple nodes using torch.distributed.launch.

    ### 1. Single node, 8 GPUs:
python -m torch.distributed.launch --nproc_per_node=8 --nnodes=1 --node_rank=0 --master_addr="127.0.0.1" --master_port=1234 train.py configs/ms1mv3_r50

### 2. Multiple nodes, each node 8 GPUs:
# Node 0:
python -m torch.distributed.launch --nproc_per_node=8 --nnodes=2 --node_rank=0 --master_addr="ip1" --master_port=1234 train.py train.py configs/ms1mv3_r50

# Node 1:
python -m torch.distributed.launch --nproc_per_node=8 --nnodes=2 --node_rank=1 --master_addr="ip1" --master_port=1234 train.py train.py configs/ms1mv3_r50

### 3. Training resnet2060 with 8 GPUs:
python -m torch.distributed.launch --nproc_per_node=8 --nnodes=1 --node_rank=0 --master_addr="127.0.0.1" --master_port=1234 train.py configs/ms1mv3_r2060.py

  12. Install PyTorch v1.8.0 for arcface_torch

    main

    Use the following commands to install PyTorch v1.8.0 on Linux or Windows, depending on your CUDA version or if you are using a CPU only.

    # CUDA 11.0
pip --default-timeout=100 install torch==1.8.0+cu111 torchvision==0.9.0+cu111 torchaudio==0.8.0 -f https://download.pytorch.org/whl/torch_stable.html

# CUDA 10.2
pip --default-timeout=100 install torch==1.8.0 torchvision==0.9.0 torchaudio==0.8.0

# CPU only
pip --default-timeout=100 install torch==1.8.0+cpu torchvision==0.9.0+cpu torchaudio==0.8.0 -f https://download.pytorch.org/whl/torch_stable.html