DOCSEARCHbyAlgolia
Submit Docs

MultiTalk Documentation

repository·main·Indexed 25 days ago

https://github.com/meigen-ai/multitalk

MultiTalk is an audio-driven multi-person conversational video generation framework that produces high-quality lip-synchronized videos from audio and text prompts. It supports single and multi-person scenarios, singing performances, and cartoon characters, with output resolutions of 480p and 720p for videos up to 15 seconds. The system includes a Gradio UI and a CLI for inference, utilizing models such as Wan2.1-I2V-14B-480P and Wav2Vec2 for audio feature extraction.

Tokens
3.5K
Snippets
7
Records
15
Agent score
83%

What's inside MultiTalk

  1. Overview of MultiTalk

    main

    MultiTalk is an audio-driven multi-person conversational video generation framework. It takes multi-stream audio input, a reference image, and a text prompt to generate videos featuring multi-person interactions with lip motions synchronized to the audio.

    Key Capabilities:

    • Realistic Conversations: Supports both single and multi-person video generation.
    • Interactive Character Control: Allows directing virtual humans via text prompts.
    • Generalization: Capable of generating singing performances and cartoon characters.
    • Resolution Flexibility: Supports 480p and 720p output at arbitrary aspect ratios.
    • Long Video Generation: Supports video generation up to 15 seconds in length.

  2. Prepare MultiTalk models

    main

    Download the required models using huggingface-cli and place them in the ./weights/ directory.

    Required Models:

    • Wan-AI/Wan2.1-I2V-14B-480P (Base model)
    • TencentGameMate/chinese-wav2vec2-base (Audio encoder)
    • hexgrad/Kokoro-82M (TTS weights)
    • MeiGen-AI/MeiGen-MultiTalk (Audio condition weights)

    Download Commands:

    huggingface-cli download Wan-AI/Wan2.1-I2V-14B-480P --local-dir ./weights/Wan2.1-I2V-14B-480P
huggingface-cli download TencentGameMate/chinese-wav2vec2-base --local-dir ./weights/chinese-wav2vec2-base
huggingface-cli download TencentGameMate/chinese-wav2vec2-base model.safetensors --revision refs/pr/1 --local-dir ./weights/chinese-wav2vec2-base
huggingface-cli download hexgrad/Kokoro-82M --local-dir ./weights/Kokoro-82M
huggingface-cli download MeiGen-AI/MeiGen-MultiTalk --local-dir ./weights/MeiGen-MultiTalk

    Linking MultiTalk weights to Wan2.1 directory: You must link or copy the MultiTalk weights into the Wan2.1 directory to ensure compatibility.

    Option 1: Linking (Recommended)

    mv weights/Wan2.1-I2V-14B-480P/diffusion_pytorch_model.safetensors.index.json weights/Wan2.1-I2V-14B-480P/diffusion_pytorch_model.safetensors.index.json_old
sudo ln -s {Absolute path}/weights/MeiGen-MultiTalk/diffusion_pytorch_model.safetensors.index.json weights/Wan2.1-I2V-14B-480P/
sudo ln -s {Absolute path}/weights/MeiGen-MultiTalk/multitalk.safetensors weights/Wan2.1-I2V-14B-480P/

    Option 2: Copying

    mv weights/Wan2.1-I2V-14B-480P/diffusion_pytorch_model.safetensors.index.json weights/Wan2.1-I2V-14B-480P/diffusion_pytorch_model.safetensors.index.json_old
cp weights/MeiGen-MultiTalk/diffusion_pytorch_model.safetensors.index.json weights/Wan2.1-I2V-14B-480P/
cp weights/MeiGen-MultiTalk/multitalk.safetensors weights/Wan2.1-I2V-14B-480P/
    huggingface-cli download Wan-AI/Wan2.1-I2V-14B-480P --local-dir ./weights/Wan2.1-I2V-14B-480P
huggingface-cli download TencentGameMate/chinese-wav2vec2-base --local-dir ./weights/chinese-wav2vec2-base
huggingface-cli download TencentGameMate/chinese-wav2vec2-base model.safetensors --revision refs/pr/1 --local-dir ./weights/chinese-wav2vec2-base
huggingface-cli download hexgrad/Kokoro-82M --local-dir ./weights/Kokoro-82M
huggingface-cli download MeiGen-AI/MeiGen-MultiTalk --local-dir ./weights/MeiGen-MultiTalk

  3. Install MultiTalk dependencies

    main

    Follow these steps to set up a conda environment and install the necessary dependencies for MultiTalk.

    1. Create Conda Environment and Install PyTorch/xformers:

      conda create -n multitalk python=3.10
conda activate multitalk
pip install torch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1 --index-url https://download.pytorch.org/whl/cu121
pip install -U xformers==0.0.28 --index-url https://download.pytorch.org/whl/cu121

    2. Install Flash-attn and related packages:

      pip install misaki[en]
pip install ninja 
pip install psutil 
pip install packaging 
pip install flash_attn==2.7.4.post1

    3. Install other dependencies:

      pip install -r requirements.txt
conda install -c conda-forge librosa

    4. Install FFmpeg: Using conda:

      conda install -c conda-forge ffmpeg

      Or using yum:

      sudo yum install ffmpeg ffmpeg-devel
    conda create -n multitalk python=3.10
conda activate multitalk
pip install torch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1 --index-url https://download.pytorch.org/whl/cu121
pip install -U xformers==0.0.28 --index-url https://download.pytorch.org/whl/cu121
pip install misaki[en]
pip install ninja 
pip install psutil 
pip install packaging 
pip install flash_attn==2.7.4.post1
pip install -r requirements.txt
conda install -c conda-forge librosa
conda install -c conda-forge ffmpeg

  4. Run MultiTalk via CLI

    main

    The MultiTalk system can be executed from the command line using app.py. This entrypoint initializes the model pipeline and launches a Gradio web interface for video generation.

    Key CLI Arguments:

    • --task: The task to run (e.g., multitalk-14B).
    • --size: The bucket size of the generated video (e.g., multitalk-480, multitalk-720).
    • --ckpt_dir: Path to the Wan checkpoint directory.
    • --wav2vec_dir: Path to the wav2vec checkpoint directory.
    • --mode: Generation mode, either clip (single chunk) or streaming (long video generation).
    • --offload_model: Whether to offload the model to CPU after each forward pass to save VRAM.
    • --ulysses_size / --ring_size: Parallelism settings for DiT.
    • --use_teacache: Enable TeaCache for faster video generation.
    • --use_apg: Enable Adaptive Projected Guidance (APG).

  5. Run Single-Person Inference

    main

    Use the following commands to generate videos with a single person.

    Standard Single GPU Inference:

    python generate_multitalk.py \
    --ckpt_dir weights/Wan2.1-I2V-14B-480P \
    --wav2vec_dir 'weights/chinese-wav2vec2-base' \
    --input_json examples/single_example_1.json \
    --sample_steps 40 \
    --mode streaming \
    --use_teacache \
    --save_file single_long_exp

    Low VRAM Inference: Set --num_persistent_param_in_dit 0 to run on hardware with limited memory.

    python generate_multitalk.py \
    --ckpt_dir weights/Wan2.1-I2V-14B-480P \
    --wav2vec_dir 'weights/chinese-wav2vec2-base' \
    --input_json examples/single_example_1.json \
    --sample_steps 40 \
    --mode streaming \
    --num_persistent_param_in_dit 0 \
    --use_teacache \
    --save_file single_long_lowvram_exp

    Multi-GPU Inference (using torchrun):

    GPU_NUM=8
torchrun --nproc_per_node=$GPU_NUM --standalone generate_multitalk.py \
    --ckpt_dir weights/Wan2.1-I2V-14B-480P \
    --wav2vec_dir 'weights/chinese-wav2vec2-base' \
    --input_json examples/single_example_1.json \
    --sample_steps 40 \
    --mode streaming \
    --save_file single_long_multigpu_exp

    Inference with TTS:

    python generate_multitalk.py \
    --ckpt_dir weights/Wan2.1-I2V-14B-480P \
    --wav2vec_dir 'weights/chinese-wav2vec2-base' \
    --input_json examples/single_example_tts_1.json \
    --sample_steps 40 \
    --mode streaming \
    --num_persistent_param_in_dit 0 \
    --use_teacache \
    --save_file single_long_lowvram_tts_exp \
    --audio_mode tts
    python generate_multitalk.py \
    --ckpt_dir weights/Wan2.1-I2V-14B-480P \
    --wav2vec_dir 'weights/chinese-wav2vec2-base' \
    --input_json examples/single_example_1.json \
    --sample_steps 40 \
    --mode streaming \
    --use_teacache \
    --save_file single_long_exp

  6. Run with Gradio UI

    main

    You can launch a Gradio web interface for MultiTalk using app.py.

    Basic Launch:

    python app.py

    Launch with Low VRAM mode:

    python app.py --num_persistent_param_in_dit 0

    Launch with FusionX LoRA and Quantization:

    python app.py \
    --quant int8 \
    --quant_dir weights/MeiGen-MultiTalk \
    --lora_dir weights/MeiGen-MultiTalk/quant_models/quant_model_int8_FusionX.safetensors \
    --sample_shift 2 \
    --num_persistent_param_in_dit 0
    python app.py --num_persistent_param_in_dit 0

  7. Run Multi-Person Inference

    main

    Use the following commands to generate videos with multiple people.

    Standard Single GPU Inference:

    python generate_multitalk.py \
    --ckpt_dir weights/Wan2.1-I2V-14B-480P \
    --wav2vec_dir 'weights/chinese-wav2vec2-base' \
    --input_json examples/multitalk_example_2.json \
    --sample_steps 40 \
    --mode streaming \
    --use_teacache \
    --save_file multi_long_exp

    Low VRAM Inference:

    python generate_multitalk.py \
    --ckpt_dir weights/Wan2.1-I2V-14B-480P \
    --wav2vec_dir 'weights/chinese-wav2vec2-base' \
    --input_json examples/multitalk_example_2.json \
    --sample_steps 40 \
    --mode streaming \
    --num_persistent_param_in_dit 0 \
    --use_teacache \
    --save_file multi_long_lowvram_exp

    Multi-GPU Inference:

    GPU_NUM=8
torchrun --nproc_per_node=$GPU_NUM --standalone generate_multitalk.py \
    --ckpt_dir weights/Wan2.1-I2V-14B-480P \
    --wav2vec_dir 'weights/chinese-wav2vec2-base' \
    --dit_fsdp --t5_fsdp --ulysses_size=$GPU_NUM \
    --input_json examples/multitalk_example_2.json \
    --sample_steps 40 \
    --mode streaming --use_teacache \
    --save_file multi_long_multigpu_exp
    python generate_multitalk.py \
    --ckpt_dir weights/Wan2.1-I2V-14B-480P \
    --wav2vec_dir 'weights/chinese-wav2vec2-base' \
    --input_json examples/multitalk_example_2.json \
    --sample_steps 40 \
    --mode streaming \
    --use_teacache \
    --save_file multi_long_exp

  9. Enable recursive VRAM management for a model

    main

    Use enable_vram_management to wrap specific layers of a torch.nn.Module with VRAM-aware wrappers. This allows for offloading weights between devices (e.g., CPU and GPU) and switching dtypes during computation to optimize memory usage.

    Parameters:

    • model: The torch.nn.Module to modify.
    • module_map: A dictionary mapping source module types (e.g., torch.nn.Linear) to their corresponding wrapper classes (e.g., AutoWrappedLinear).
    • module_config: A dictionary of configuration arguments passed to the wrapper's __init__ (e.g., offload_dtype, offload_device, onload_dtype, onload_device, computation_dtype, computation_device).
    • max_num_param (optional): A threshold for the number of parameters. If the cumulative parameter count exceeds this value, the overflow_module_config is used instead of module_config.
    • overflow_module_config (optional): Configuration arguments used for modules that exceed the max_num_param threshold.

  10. Use Wav2Vec2Model for audio feature extraction

    main

    The Wav2Vec2Model class extends the Hugging Face Wav2Vec2Model to provide specialized audio feature extraction. It includes methods for extracting features with linear interpolation to a specific sequence length and encoding those features using a transformer encoder.

    Key Methods

    • forward(input_values, seq_len, attention_mask=None, mask_time_indices=None, output_attentions=None, output_hidden_states=None, return_dict=None): Performs the full forward pass, including feature extraction, interpolation to seq_len, projection, and encoding.
    • feature_extract(input_values, seq_len): Extracts features from raw input_values and applies linear interpolation to match the target seq_len.
    • encode(extract_features, attention_mask=None, mask_time_indices=None, output_attentions=None, output_hidden_states=None, return_dict=None): Takes already extracted features and passes them through the projection and encoder layers.

  11. Wav2Vec2Model.feature_extract

    main

    The feature_extract method processes raw audio input into interpolated feature vectors. It uses the internal feature_extractor, transposes the result, and applies linear_interpolation to ensure the output matches the requested seq_len.

    def feature_extract(
    self,
    input_values,
    seq_len,
):

  12. AutoWrappedQLinear wrapper for quantized VRAM management

    main

    A wrapper for optimum.quanto.nn.qlinear.QLinear designed for quantized layers. It manages the movement of quantized weights between devices to optimize VRAM usage.

    Initialization Arguments:

    • module: The original qlinear.QLinear instance.
    • offload_device: The device where weights reside when not in active computation.
    • onload_device: The device where weights are moved for computation.
    • computation_device: The device used for the linear operation.