IMTalker: Audio- and Video-Driven Talking Face Generation

For manual setup or offline use, download the following components from Hugging Face and place them in the ./checkpoints directory. Ensure the directory structure matches the requirement below.

./checkpoints
├── renderer.ckpt                     # The main renderer
├── generator.ckpt                    # The main generator
└── wav2vec2-base-960h/               # Audio encoder folder
    ├── config.json
    ├── pytorch_model.bin
    └── ...

Follow these guidelines to improve the quality of generated talking faces:

1. Input Image: Use images where the head is the primary subject. The model does not support full-body generation. It generates at a fixed 512×512 resolution; prioritize facial clarity over high resolution.
2. Audio: Use English audio for the best lip-synchronization and naturalness, as the model was primarily trained on English datasets.
3. Background: Use source images with solid colored or blurred (bokeh) backgrounds to minimize visual artifacts or jitter.

To train the generator for speech-driven animation, prepare a dataset containing pre-extracted features: motion (.pt), audio (.npy), smirk (.pt), and gaze (.npy).

python generator/train.py \
    --dataset_pat /path/to/generator_dataset \
    --exp_name generator_exp \
    --batch_size 16 \
    --iter 5000000 \
    --lr 1e-4

To train the renderer, prepare a dataset where videos are cropped to 512×512 resolution and landmarks are extracted. The directory must contain video_frame (organized by video ID) and lmd (landmark text files).

python renderer/train.py \
    --dataset_path /path/to/renderer_dataset \
    --exp_name renderer_exp \
    --batch_size 4 \
    --iter 7000000 \
    --lr 1e-4

To install IMTalker, first set up a Conda environment with Python 3.10 and install the required PyTorch and FFmpeg dependencies. Then, clone the repository and install the remaining requirements via pip.

# 1. Environment Setup
conda create -n IMTalker python=3.10
conda activate IMTalker
pip install torch==2.3.1 torchvision==0.18.1 torchaudio==2.3.1 --index-url https://download.pytorch.org/whl/cu121
conda install -c conda-forge ffmpeg 

# 2. Install with pip
git clone https://github.com/cbsjtu01/IMTalker.git
cd IMTalker
pip install -r requirement.txt

The fastest way to use IMTalker is via the Gradio web interface. Running the app.py script will automatically download the necessary pretrained models into the ./checkpoints directory if they are not already present.

python app.py

The application includes an automatic download mechanism. When ensure_checkpoints() is called, it checks the checkpoints/ directory for required files from the cbsjtu01/IMTalker Hugging Face repository. It verifies existence and ensures files are larger than 1KB before attempting a download via hf_hub_download.

Required files include:

• config.yaml
• renderer.ckpt
• generator.ckpt
• wav2vec2-base-960h/ (config, pytorch_model, and preprocessor configs)

To achieve optimal results with IMTalker, follow these guidelines:

1. Input Image: Use images where the person's head is the primary subject. The model is trained on facial data and does not support full-body generation. The pipeline defaults to a 512x512 resolution; prioritize facial clarity over high resolution.
2. Audio: The model was primarily trained on English datasets. Use English audio for the best lip-synchronization and naturalness.
3. Background: Use source images with solid colored or blurred (bokeh) backgrounds. Complex backgrounds can cause visual artifacts or jitter in the generated video.

Generate a talking face by providing a source portrait image and a driving video file using the renderer/inference.py script.

python renderer/inference.py \
    --source_path "./assets/source_image.jpg" \
    --driving_path "./assets/driving_video.mp4" \
    --save_path "./results/" \
    --renderer_path "./checkpoints/renderer.ckpt" \
    --crop

Generate a talking face by providing a source portrait image and an audio file using the generator/generate.py script.

python generator/generate.py \
    --ref_path "./assets/source_image.jpg" \
    --aud_path "./assets/input_audio.wav" \
    --res_dir "./results/" \
    --generator_path "./checkpoints/generator.ckpt" \
    --renderer_path "./checkpoints/renderer.ckpt" \
    --a_cfg_scale 2 \
    --crop

The AppConfig class manages the runtime settings for the IMTalker service. It automatically detects if a CUDA-capable GPU is available and sets the device accordingly. Key configuration parameters include:

• renderer_path & generator_path: Paths to the model checkpoints.
• wav2vec_model_path: Path to the Wav2Vec2 pre-trained model.
• input_size: The target resolution for generation (default is 512).
• fps: Frames per second for the output video.
• sampling_rate: Audio sampling rate (default 16000).
• nfe: Number of function evaluations (steps) for the generator.
• a_cfg_scale: Classifier-Free Guidance (CFG) scale.
• crop: Boolean flag to enable/disable automatic face cropping.