DOCSEARCHbyAlgolia
Submit Docs

Live Avatar

repository·main·Indexed 25 days ago

https://github.com/alibaba-quark/liveavatar

A framework for real-time, streaming, infinite-length audio-driven avatar video generation. It utilizes a 14B-parameter diffusion model and supports block-wise autoregressive processing for long video sequences. The project includes a face detector adapted from face-detection-pytorch and supports both single-GPU and multi-GPU inference modes.

Tokens
1.2K
Snippets
4
Records
6
Agent score
32%

What's inside Live Avatar

  2. License and Usage Terms

    main
    The liveavatar project is a research preview. The majority of the project is released under the Apache 2.0 license. The base model, Wan, is also released under the Apache 2.0 license. For potential violations or inquiries, contact jmliu1217@gmail.com.

  3. Install Live Avatar

    main

    Follow these steps to set up the environment for Live Avatar:

    1. Create a Conda environment:

      conda create -n liveavatar python=3.10 -y
conda activate liveavatar

    2. Install CUDA Dependencies (optional):

      conda install nvidia/label/cuda-12.4.1::cuda -y
conda install -c nvidia/label/cuda-12.4.1 cudatoolkit -y

    3. Install PyTorch & Flash Attention: Install PyTorch first:

      pip install torch==2.8.0 torchvision==0.23.0 --index-url https://download.pytorch.org/whl/cu128

      Then install the appropriate Flash Attention version:

      • For NVIDIA Hopper architecture (H800/H200, etc.), use FlashAttention 3:
        pip install flash_attn_3 --find-links https://windreamer.github.io/flash-attention3-wheels/cu128_torch280 --extra-index-url https://download.pytorch.org/whl/cu128
      • For other architectures, use FlashAttention 2:
        pip install flash-attn==2.8.3 --no-build-isolation

    4. Install Python Requirements:

      pip install -r requirements.txt

    5. Install FFMPEG:

      apt-get update && apt-get install -y ffmpeg
    conda create -n liveavatar python=3.10 -y
conda activate liveavatar
# ... follow subsequent steps in README

  4. Download Live Avatar Models

    main

    Download the required pretrained checkpoints and place them in the ./ckpt/ directory.

    If you are in mainland China, set the environment variable first: export HF_ENDPOINT=https://hf-mirror.com.

    Use the following commands to download the models:

    pip install "huggingface_hub[cli]"
huggingface-cli download Wan-AI/Wan2.2-S2V-14B --local-dir ./ckpt/Wan2.2-S2V-14B
huggingface-cli download Quark-Vision/Live-Avatar --local-dir ./ckpt/LiveAvatar

    Expected Directory Structure:

    ckpt/
├── Wan2.2-S2V-14B/          # Base model
│   ├── config.json
│   ├── diffusion_pytorch_model-*.safetensors
│   └── ...
└── LiveAvatar/              # Our LoRA model
    ├── liveavatar.safetensors
    └── ...
    huggingface-cli download Wan-AI/Wan2.2-S2V-14B --local-dir ./ckpt/Wan2.2-S2V-14B
huggingface-cli download Quark-Vision/Live-Avatar --local-dir ./ckpt/LiveAvatar

  5. Run Single-GPU Inference

    main

    Use this mode for offline generation on a single GPU with at least 80GB VRAM.

    Commands:

    • CLI Inference: bash infinite_inference_single_gpu.sh
    • Gradio Web UI: bash gradio_single_gpu.sh

    Tips:

    • OOM Errors: If you encounter Out-of-Memory errors in the Gradio Web UI, try lowering the resolution using the size parameter.
    • Quality vs. Performance: By default, enable_online_decode is set to false to avoid performance degradation from CPU offloading. If you are generating extremely long videos and need higher quality, add the --enable_online_decode flag to your command.
    # CLI Inference
bash infinite_inference_single_gpu.sh

# Gradio Web UI
bash gradio_single_gpu.sh

  6. Run Real-time Multi-GPU Inference

    main

    Use this mode for real-time streaming. This currently requires five GPUs and at least 80GB VRAM per GPU.

    Commands:

    • CLI Inference: bash infinite_inference_multi_gpu.sh
    • Gradio Web UI: bash gradio_multi_gpu.sh

    Configuration Options (via Environment Variables):

    • ENABLE_COMPILE: Set to true to enable compilation. This causes a long initial wait but significantly improves performance for long streaming videos. Set to false for quick tests.
    • ENABLE_FP8: Set to true to enable FP8 quantization. This provides notable VRAM savings (enabling inference on 48GB GPUs) and modest performance gains, though it may cause slight quality degradation.

    Input Details: The model generates videos from audio input combined with a reference image and an optional text prompt. The size parameter controls the generated video area (aspect ratio follows the input image). The --num_clip parameter controls the number of video clips generated (useful for quick previews).

    # CLI Inference
bash infinite_inference_multi_gpu.sh

# Gradio Web UI
bash gradio_multi_gpu.sh