InfiniteTalk Documentation

InfiniteTalk is an audio-driven video generation model designed for sparse-frame video dubbing. It supports two primary modes of operation:

1. Video-to-Video: Given an input video and an audio track, it synthesizes a new video with accurate lip synchronization while aligning head movements, body posture, and facial expressions with the audio.
2. Image-to-Video: Uses an image and an audio track as input to generate a talking video.

Key capabilities include infinite-length video generation, consistent identity preservation, and improved stability in body/hand movements compared to previous methods like MultiTalk.

Follow these steps to set up the environment using Conda and Pip:

1. Create Conda environment and install PyTorch/xformers:

conda create -n multitalk python=3.10
conda activate multitalk
pip install torch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1 --index-url https://download.pytorch.org/whl/cu121
pip install -U xformers==0.0.28 --index-url https://download.pytorch.org/whl/cu121

2. Install Flash-attn and related packages:

pip install misaki[en]
pip install ninja 
pip install psutil 
pip install packaging
pip install wheel
pip install flash_attn==2.7.4.post1

3. Install other dependencies:

pip install -r requirements.txt
conda install -c conda-forge librosa

4. Install FFmpeg: Using Conda:

conda install -c conda-forge ffmpeg

Or using yum:

sudo yum install ffmpeg ffmpeg-devel

Download the required weights using huggingface-cli. You need the base model, the audio encoder, and the InfiniteTalk weights.

Required models:

• Wan-AI/Wan2.1-I2V-14B-480P (Base model)
• TencentGameMate/chinese-wav2vec2-base (Audio encoder)
• MeiGen-AI/InfiniteTalk (Audio condition weights)

huggingface-cli download Wan-AI/Wan2.1-I2V-14B-480P --local-dir ./weights/Wan2.1-I2V-14B-480P
huggingface-cli download TencentGameMate/chinese-wav2vec2-base --local-dir ./weights/chinese-wav2vec2-base
huggingface-cli download TencentGameMate/chinese-wav2vec2-base model.safetensors --revision refs/pr/1 --local-dir ./weights/chinese-wav2vec2-base
huggingface-cli download MeiGen-AI/InfiniteTalk --local-dir ./weights/InfiniteTalk

To run inference on a single GPU for 480P streaming video, use the following command. Ensure you have downloaded the weights to the specified directories.

python generate_infinitetalk.py \
    --ckpt_dir weights/Wan2.1-I2V-14B-480P \
    --wav2vec_dir 'weights/chinese-wav2vec2-base' \
    --infinitetalk_dir weights/InfiniteTalk/single/infinitetalk.safetensors \
    --input_json examples/single_example_image.json \
    --size infinitetalk-480 \
    --sample_steps 40 \
    --mode streaming \
    --motion_frame 9 \
    --save_file infinitetalk_res

Launch a Gradio web interface for interactive inference. You can use either single-person or multi-person weights.

python app.py \
    --ckpt_dir weights/Wan2.1-I2V-14B-480P \
    --wav2vec_dir 'weights/chinese-wav2vec2-base' \
    --infinitetalk_dir weights/InfiniteTalk/single/infinitetalk.safetensors \
    --num_persistent_param_in_dit 0 \
    --motion_frame 9

For multi-GPU inference, use torchrun with FSDP and Ulysses parallelism. Set GPU_NUM to your available GPU count.

GPU_NUM=8
torchrun --nproc_per_node=$GPU_NUM --standalone generate_infinitetalk.py \
    --ckpt_dir weights/Wan2.1-I2V-14B-480P \
    --wav2vec_dir 'weights/chinese-wav2vec2-base' \
    --infinitetalk_dir weights/InfiniteTalk/single/infinitetalk.safetensors \
    --dit_fsdp --t5_fsdp \
    --ulysses_size=$GPU_NUM \
    --input_json examples/single_example_image.json \
    --size infinitetalk-480 \
    --sample_steps 40 \
    --mode streaming \
    --motion_frame 9 \
    --save_file infinitetalk_res_multigpu

To use LoRA (like FusionX which requires 8 steps or lightx2v which requires 4), provide the --lora_dir and adjust the guide scales. Note that when using LoRA, the recommended --sample_text_guide_scale is 1.0 and --sample_audio_guide_scale is 2.0.

python generate_infinitetalk.py \
    --ckpt_dir weights/Wan2.1-I2V-14B-480P \
    --wav2vec_dir 'weights/chinese-wav2vec2-base' \
    --infinitetalk_dir weights/InfiniteTalk/single/infinitetalk.safetensors \
    --lora_dir weights/Wan2.1_I2V_14B_FusionX_LoRA.safetensors \
    --input_json examples/single_example_image.json \
    --lora_scale 1.0 \
    --size infinitetalk-480 \
    --sample_text_guide_scale 1.0 \
    --sample_audio_guide_scale 2.0 \
    --sample_steps 8 \
    --mode streaming \
    --motion_frame 9 \
    --sample_shift 2 \
    --num_persistent_param_in_dit 0 \
    --save_file infinitetalk_res_lora

InfiniteTalk supports two TTS modes via the Kokoro pipeline:

1. Single Speaker TTS (process_tts_single): Generates audio from a single text prompt using a specified voice tensor. It saves the result as s1.wav in the provided directory.
2. Multi-Speaker TTS (process_tts_multi): Parses text containing speaker tags in the format (s1) text (s2) text. It generates separate audio tracks for each speaker and a combined sum.wav track. This allows for complex dialogue generation within a single prompt.

Use enable_vram_management to wrap specific layers in the model with VRAM-aware wrappers. This allows for offloading weights to different devices or dtypes to manage memory usage.

To use this, you must provide a module_map that maps existing module types (e.g., torch.nn.Linear) to their corresponding wrapper classes (e.g., AutoWrappedLinear).

Arguments:

• model: The torch.nn.Module to modify.
• module_map: A dictionary mapping source module types to wrapper classes.
• module_config: A dictionary of keyword arguments passed to the wrapper's __init__.
• max_num_param (optional): A threshold for parameter count. If the cumulative parameter count exceeds this, overflow_module_config is used instead of module_config.
• overflow_module_config (optional): Configuration used when the parameter threshold is exceeded.