OmniAvatar Documentation

The --input_file (e.g., examples/infer_samples.txt) controls the character behavior. Each line must follow the format: [prompt]@@[img_path]@@[audio_path]

Prompting Best Practices:

• Use the structure: [Description of first frame] - [Description of human behavior] - [Description of background (optional)].
• Guidance Scales: The recommended range for both prompt and audio cfg is [4-6]. Increasing audio cfg improves lip-sync consistency.
• Guidance Control: guidance_scale controls prompts. To control audio guidance separately, use audio_scale=3.

To set up OmniAvatar, clone the repository and install the required Python dependencies. It is recommended to install flash_attn to accelerate attention computation.

git clone https://github.com/Omni-Avatar/OmniAvatar
cd OmniAvatar

# Install PyTorch dependencies
pip install torch==2.4.0 torchvision==0.19.0 torchaudio==2.4.0 --index-url https://download.pytorch.org/whl/cu124

# Install requirements
pip install -r requirements.txt

# Optional: accelerate attention computation
pip install flash_attn

Use the --hp flag to pass hyperparameters to scripts/inference.py to optimize speed, quality, or VRAM usage.

Speed and Quality

• Steps: num_steps in range [20-50] (higher is better quality).
• TeaCache: To speed up, set tea_cache_l1_thresh in range [0.05-0.15] (e.g., 0.14).
• Multi-GPU: Set sp_size to your GPU count (e.g., sp_size=8).

Memory Reduction

To reduce VRAM usage, enable FSDP and set num_persistent_param_in_dit.

Example optimized command:

torchrun --standalone --nproc_per_node=8 scripts/inference.py \
  --config configs/inference.yaml \
  --input_file examples/infer_samples.txt \
  --hp=sp_size=8,max_tokens=30000,guidance_scale=4.5,overlap_frame=13,num_steps=25,use_fsdp=True,tea_cache_l1_thresh=0.14,num_persistent_param_in_dit=7000000000

OmniAvatar requires base models (Wan2.1), OmniAvatar LoRA/audio weights, and a Wav2Vec audio encoder. You can download these using the huggingface-cli into a pretrained_models directory.

mkdir pretrained_models
pip install "huggingface_hub[cli]"

# Download 14B components
huggingface-cli download Wan-AI/Wan2.1-T2V-14B --local-dir ./pretrained_models/Wan2.1-T2V-14B
huggingface-cli download facebook/wav2vec2-base-960h --local-dir ./pretrained_models/wav2vec2-base-960h
huggingface-cli download OmniAvatar/OmniAvatar-14B --local-dir ./pretrained_models/OmniAvatar-14B

Run inference using torchrun. Currently, only 480p resolution is supported. You can choose between the 14B and 1.3B model configurations.

14B Inference:

torchrun --standalone --nproc_per_node=1 scripts/inference.py --config configs/inference.yaml --input_file examples/infer_samples.txt

1.3B Inference:

torchrun --standalone --nproc_per_node=1 scripts/inference.py --config configs/inference_1.3B.yaml --input_file examples/infer_samples.txt

The following parameters can be passed via the --hp flag to tune the inference process: