repository·main·Indexed 23 days ago
https://github.com/fantasy-amap/fantasy-talkingA framework for generating realistic talking portraits by synthesizing coherent motion from audio and image inputs. It utilizes the Wan2.1-I2V-14B-720P base model and Wav2Vec audio encoder. The documentation covers installation, inference via infer.py, Gradio demo setup, and detailed optimization tips for ComfyUI workflows, including Audio CFG Schedule configuration to improve lip-sync accuracy and strategies for long-form video generation.
Generating long sequences with Wan 2.1 can lead to motion drift or detail inconsistency. To maintain consistency in long-form video, use the sliding-window approach: denoise in overlapping chunks and fuse latent features in the overlap region.
Alternative methods include:
The community version of the FantasyTalking ComfyUI workflow often suffers from poor lip-syncing because the default end_percent in the Audio CFG Schedule node is set too low (0.1). This means audio conditioning is only applied during the first 10% of denoising steps.
To achieve results closer to the official implementation (which applies audio CFG across all steps), you should increase the end_percent value.
Recommended Setting:
end_percent $\ge$ 0.7 in the Audio CFG Schedule node.Generating long sequences with Wan2.1 can lead to motion drift and loss of character consistency.
Recommended Strategies:
For a reference implementation, see the kijai long video generation workflow.
To use FantasyTalking, clone the repository and install the required Python dependencies. Ensure you have torch >= 2.0.0 installed. You can optionally install flash_attn to accelerate attention computation.
git clone https://github.com/Fantasy-AMAP/fantasy-talking.git
cd fantasy-talking
# Install dependencies
pip install -r requirements.txt
# Optional: Install flash_attn for faster attention computation
pip install flash_attnFor a balanced generation in ComfyUI, use the following hyperparameter guidelines:
| Parameter | Recommended Value | Purpose |
|---|---|---|
end_percent | $\ge$ 0.7 | Controls duration of audio conditioning (crucial for lip-sync) |
Sampling Steps | $\ge$ 30 | Ensures sufficient denoising quality |
CFG Scale | 3.0 - 7.0 | Balances fidelity and creativity |
audio_cfg_scale | 3.0 - 7.0 | Controls influence of audio on video generation |
prompt_cfg_scale | 3.0 - 7.0 | Controls influence of text prompts on video generation |
FantasyTalking requires three main model components: the base model (Wan2.1-I2V-14B-720P), an audio encoder (Wav2Vec), and the FantasyTalking audio-conditional weights. You can download these using either huggingface-cli or modelscope-cli.
# Using huggingface-cli
pip install "huggingface_hub[cli]"
huggingface-cli download Wan-AI/Wan2.1-I2V-14B-720P --local-dir ./models/Wan2.1-I2V-14B-720P
huggingface-cli download facebook/wav2vec2-base-960h --local-dir ./models/wav2vec2-base-960h
huggingface-cli download acvlab/FantasyTalking fantasytalking_model.ckpt --local-dir ./models
# OR using modelscope-cli
pip install modelscope
modelscope download Wan-AI/Wan2.1-I2V-14B-720P --local_dir ./models/Wan2.1-I2V-14B-720P
modelscope download AI-ModelScope/wav2vec2-base-960h --local_dir ./models/wav2vec2-base-960h
modelscope download amap_cvlab/FantasyTalking fantasytalking_model.ckpt --local_dir ./modelsIn the Community Edition ComfyUI workflow, the default end_percent in the Audio CFG Schedule node is often set to 0.1, which causes inaccurate lip-sync because audio conditioning is only applied during the first 10% of denoising steps.
To achieve lip-sync quality comparable to the official implementation, set end_percent to 0.7 or higher (the official implementation uses 1.0). This extends the range of denoising steps influenced by audio conditioning.
To install FantasyTalking, clone the repository and install the required dependencies. Ensure you have torch >= 2.0.0 installed. You can optionally install flash_attn to accelerate attention computation.
git clone https://github.com/Fantasy-AMAP/fantasy-talking.git
cd fantasy-talking
# Install dependencies
pip install -r requirements.txt
# Optional: accelerate attention computation
pip install flash_attnThe project includes a Gradio-based web interface for interactive video generation. When running app.py as a script, it launches a local web server that allows you to upload images and audio, adjust sliders for CFG scales and weights, and view the resulting video directly in the browser.
To launch the demo, execute the script directly:
python app.pyBecause FantasyTalking is based on an I2V (Image-to-Video) model, the first frame may suffer from color drift or blur. High audio conditioning can amplify these artifacts.
To mitigate this, you can trade off lip-sync precision for visual stability by:
end_percent to approximately 0.7).audio_cfg_scale.Because FantasyTalking is based on Image-to-Video (I2V) models, the first frame may exhibit color drift or blur. This is exacerbated by the audio conditioning training.
Solutions:
end_percent (e.g., to 0.7) or decrease the audio_cfg_scale. This makes the model rely more on the base model's first-frame reference, though it may slightly reduce lip-sync precision.Frame Interpolation or Deblur nodes after generation.To launch a local web interface for the model, install gradio and spaces, then run app.py.
pip install gradio spaces
python app.py