LiveTalk

LiveTalk is an end-to-end pipeline for real-time talking avatars. It consists of several key components:

• Thinker / Talker (Audio LM): A streaming audio language model that processes user text or audio to produce real-time speech responses.
• Performer (Few-Step Diffusion): A 4-step causal video diffusion model that generates video in block-wise AR fashion (3 latent frames per block). It is conditioned on streaming audio, a reference avatar image, and motion-focused text prompts.
• KV Cache & Streaming: Uses a prefilled KV cache from previous blocks to maintain temporal coherence while enabling low-latency, block-by-block video streaming synchronized to the audio.
• Long-Form Identity Preservation: Employs Anchor-Heavy Identity Sinks (AHIS), which reserve part of the KV window as fixed "identity anchors" while using a smaller rolling window for context. This stabilizes the avatar's appearance during long interactions.
• Parallel Denoising & Decoding: Diffusion denoising and VAE decoding are pipelined so that generation stays ahead of playback, ensuring real-time rendering without stalls.

LiveTalk requires three specific model checkpoints to be downloaded into the pretrained_checkpoints/ directory. Use huggingface-cli to download them:

1. Wan2.1 base model: Wan-AI/Wan2.1-T2V-1.3B
2. LiveTalk model checkpoint: GAIR/LiveTalk-1.3B-V0.1
3. Wav2Vec2 model: facebook/wav2vec2-base-960h (for audio processing)

The resulting directory structure should look like this:

.livetalk
|-- OmniAvatar
|-- pretrained_checkpoints
|   |-- LiveTalk-1.3B-V0.1
|   |-- Wan2.1-T2V-1.3B
|   |-- wav2vec2

# Download Wan2.1 base model
huggingface-cli download Wan-AI/Wan2.1-T2V-1.3B --local-dir-use-symlinks False --local-dir pretrained_checkpoints/Wan2.1-T2V-1.3B

# Download LiveTalk model checkpoint
huggingface-cli download GAIR/LiveTalk-1.3B-V0.1 --local-dir-use-symlinks False --local-dir pretrained_checkpoints/LiveTalk-1.3B-V0.1

# Download Wav2Vec2 model for audio processing
huggingface-cli download facebook/wav2vec2-base-960h --local-dir-use-symlinks False --local-dir pretrained_checkpoints/wav2vec2

To set up LiveTalk, you must clone the main repository, clone the OmniAvatar dependency, apply specific patches, and set up a Conda environment.

Prerequisites:

• NVIDIA GPU with at least 24 GB memory (e.g., RTX 4090, A800, H800).
• Linux operating system.
• 64 GB RAM.

Installation Steps:

1. Clone the repositories:

   git clone https://github.com/GAIR-NLP/livetalk.git
   cd livetalk
   git clone https://github.com/Omni-Avatar/OmniAvatar

2. Apply patches to OmniAvatar:

   bash scripts/add_patch.sh

3. Create and configure the Conda environment:

   conda create -n livetalk python=3.10 -y
   conda activate livetalk
   pip install -r requirements.txt
   conda install -c conda-forge ffmpeg
   pip install https://github.com/Dao-AILab/flash-attention/releases/download/v2.8.3/flash_attn-2.8.3+cu12torch2.8cxx11abiFALSE-cp310-cp310-linux_x86_64.whl
   python setup.py develop