MimicMotion Documentation

You must manually download the model weights and organize them into a models/ directory. If you have connection issues with Hugging Face, set the environment variable export HF_ENDPOINT=https://hf-mirror.com before downloading.

Note: The SVD model (stabilityai/stable-video-diffusion-img2vid-xt-1-1) will be downloaded automatically during runtime.

cd MimicMotions/
mkdir models

# 1. Download DWPose pretrained models
mkdir -p models/DWPose
wget https://huggingface.co/yzd-v/DWPose/resolve/main/yolox_l.onnx?download=true -O models/DWPose/yolox_l.onnx
wget https://huggingface.co/yzd-v/DWPose/resolve/main/dw-ll_ucoco_384.onnx?download=true -O models/DWPose/dw-ll_ucoco_384.onnx

# 2. Download MimicMotion checkpoint
wget -P models/ https://huggingface.co/tencent/MimicMotion/resolve/main/MimicMotion_1-1.pth

To set up the MimicMotion environment, use Conda to create an environment from the provided environment.yaml file. It is recommended to use Python 3+ and Torch 2.x. The setup has been validated with an Nvidia V100 GPU.

conda env create -f environment.yaml
conda activate mimicmotion

Run inference using the inference.py script by providing a configuration file (e.g., configs/test.yaml). You can modify the YAML configuration file to adjust parameters according to your needs.

If you encounter insufficient memory, you can try setting the environment variable PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:256 to manage GPU memory more efficiently.

python inference.py --inference_config configs/test.yaml

MimicMotion's resource requirements depend on the model version and hardware:

• 72-frame model (Version 1.1): Requires approximately 16GB VRAM (e.g., RTX 4060 Ti). On an RTX 4090, a 35s demo video takes about 20 minutes.
• 16-frame U-Net model: Minimum VRAM requirement is 8GB.
• VAE Decoder: Demands 16GB VRAM. If you have limited VRAM, you have the option to run the VAE decoder on the CPU.

Tips for low memory:

• Reduce the number of frames in your configuration.
• Set PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:256.
• Run the VAE decoder on CPU.

Ensure your models directory is organized as follows for the inference to work correctly:

models/
├── DWPose
│   ├── dw-ll_ucoco_384.onnx
│   └── yolox_l.onnx
└── MimicMotion_1-1.pth