UniAnimate: Unified Video Diffusion for Human Image Animation

UniAnimate is a framework for consistent human image animation using unified video diffusion models. It is designed to address limitations in existing techniques by:

1. Reducing Optimization Burden: It maps reference images, posture guidance, and noise video into a common feature space using a unified video diffusion model, eliminating the need for an extra reference model.
2. Enabling Long-term Generation: It uses a unified noise input (supporting both random noised input and first-frame conditioned input) to facilitate the generation of long-term videos, including consistent one-minute videos via iterative first-frame conditioning.
3. Efficient Temporal Modeling: It utilizes an alternative temporal modeling architecture based on state space models (SSM) instead of standard temporal Transformers to handle long sequences more efficiently.

For high-memory GPUs (e.g., A100), you can accelerate long video inference by enabling multiple segments parallel denoising. This is controlled by the context_batch_size parameter in the configuration file.

To enable this, modify configs/UniAnimate_infer_long.yaml and set context_batch_size to a value greater than 1 (e.g., 4).

# configs/UniAnimate_infer_long.yaml
context_batch_size: 4

Pose alignment rescales the target pose sequence to match the pose of the reference image. This is a critical step for consistent animation.

Requirements:

• To run pose alignment on GPU, install onnxruntime-gpu==1.13.1.

Best Practice:

• Ensure the first frame of the target pose sequence contains the entire face and pose (including hands and feet). This helps the model calculate a more accurate scale coefficient for better video generation results.

Note: If you are running the provided demo videos, this step can be skipped as processed poses are already provided in data/saved_pose.

# Example: Aligning pose for a specific reference image
python run_align_pose.py --ref_name data/images/musk.jpg --source_video_paths data/videos/source_video.mp4 --saved_pose_dir data/saved_pose/musk

You can generate video clips using inference.py by pointing to a configuration file.

Standard Resolution (768x512)

Generates 32-frame clips. Requires ~12G GPU memory.

python inference.py --cfg configs/UniAnimate_infer.yaml

High Resolution (1216x768)

To use higher resolution, modify resolution: [512, 768] to resolution: [768, 1216] in configs/UniAnimate_infer.yaml. Requires ~21G GPU memory.

python inference.py --cfg configs/UniAnimate_infer.yaml

Customizing Length and Memory

If your GPU memory is limited, you can reduce the number of frames by changing max_frames: 32 to a lower value (e.g., 24, 16, or 8) in configs/UniAnimate_infer.yaml. The model is compatible with these shorter lengths.

# Generate standard 32-frame clips
python inference.py --cfg configs/UniAnimate_infer.yaml

You can significantly reduce GPU memory requirements by offloading the CLIP and VAE components to the CPU and explicitly using torch.float16.

By setting CPU_CLIP_VAE: True in configs/UniAnimate_infer.yaml, generating a 32x768x512 video clip requires approximately 12G of GPU memory.

# configs/UniAnimate_infer.yaml
CPU_CLIP_VAE: True

To set up the UniAnimate environment, clone the repository and create a Conda environment with Python 3.9. Install PyTorch with CUDA 11.8 support and then install the remaining requirements via pip.

Windows Users: Refer to this issue for specific dependency installation instructions.

Troubleshooting: If you encounter the error The shape of the 2D attn_mask is torch.Size([77, 77]), but should be (1, 1)., refer to this issue.

git clone https://github.com/ali-vilab/UniAnimate.git
cd UniAnimate
conda create -n UniAnimate python=3.9
conda activate UniAnimate
conda install pytorch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 pytorch-cuda=11.8 -c pytorch -c nvidia
pip install -r requirements.txt

Download the model weights using modelscope. After downloading, move the files from the nested directory to the top-level checkpoints/ directory so the model can find them.

The expected structure in ./checkpoints/ is:

• dw-ll_ucoco_384.onnx
• open_clip_pytorch_model.bin
• unianimate_16f_32f_non_ema_223000.pth
• v2-1_512-ema-pruned.ckpt
• yolox_l.onnx

!pip install modelscope
from modelscope.hub.snapshot_download import snapshot_download
model_dir = snapshot_download('iic/unianimate', cache_dir='checkpoints/')

mv ./checkpoints/iic/unianimate/* ./checkpoints/

If you use the UniAnimate codebase for research, please cite the following paper:

@article{wang2024unianimate,
      title={UniAnimate: Taming Unified Video Diffusion Models for Consistent Human Image Animation},
      author={Wang, Xiang and Zhang, Shiwei and Gao, Changxin and Wang, Jiayu and Zhou, Xiaoqiang and Zhang, Yingya and Yan, Luxin and Sang, Nong},
      journal={arXiv preprint arXiv:2406.01188},
      year={2024}
}

To achieve better appearance preservation (such as maintaining the background), especially during long video generation, you can use a noise prior.

In the inference script tools/inferences/inference_unianimate_long_entrance.py, the noise is sampled using the noise_prior_value from the configuration:

noise = diffusion.q_sample(random_ref_frame.clone(), getattr(cfg, "noise_prior_value", 939), noise=noise)

# Example logic in tools/inferences/inference_unianimate_long_entrance.py
noise = diffusion.q_sample(random_ref_frame.clone(), getattr(cfg, "noise_prior_value", 939), noise=noise)