AnimateDiff

AnimateDiff learns transferable motion priors that can be applied to various Stable Diffusion models through a three-stage pipeline:

1. Domain Adapter (e.g., v3_sd15_adapter.ckpt): Trained to alleviate negative effects (like watermarks) in training data and to help disentangle motion from spatial appearance. It can be removed during inference or adjusted using a LoRA scaler.
2. Motion Module (e.g., v3_sd15_mm.ckpt): The core component trained to learn real-world motion patterns from video data.
3. MotionLoRA (e.g., v2_lora_ZoomIn.ckpt): An optional stage to efficiently adapt the motion module to specific patterns like camera zooming or rolling.

To use AnimateDiff, clone the repository and install the required dependencies using pip.

Note on Branches:

• Use the main branch for Stable Diffusion V1.5.
• Use the sdxl-beta branch for Stable Diffusion XL.

Checkpoint downloading is handled automatically by the scripts, so the first execution may take longer.

git clone https://github.com/guoyww/AnimateDiff.git
cd AnimateDiff

pip install -r requirements.txt

For an easier user experience, you can launch a Gradio web application. By default, the interface will be available at http://localhost:7860.

python -u app.py

To prepare a dataset for training, download the video files and their corresponding .csv annotations (e.g., from WebVid10M).

Important Requirements:

• By default, the exemplar training script requires all video files to be stored in a single folder.
• If you need to use a different directory structure, you must modify animatediff/data/dataset.py.

Training is executed using torchrun. Depending on your goal, use the corresponding configuration file.

To finetune the UNet's image layers: Use the image_finetune.yaml configuration.

To train motion modules: Use the standard training.yaml configuration.

# Finetune UNet image layers
torchrun --nnodes=1 --nproc_per_node=1 train.py --config configs/training/v1/image_finetune.yaml

# Train motion modules
torchrun --nnodes=1 --nproc_per_node=1 train.py --config configs/training/v1/training.yaml

AnimateDiff can be used via several third-party interfaces and platforms:

• Stable Diffusion WebUI: sd-webui-animatediff
• ComfyUI: ComfyUI-AnimateDiff-Evolved
• Google Colab: AnimateDiff-colab

After preparing your dataset, you must update the data paths in the configuration .yaml files located in the configs/training/ directory.

Update the train_data section with your specific paths:

• csv_path: The path to your .csv annotation file.
• video_folder: The path to the folder containing your video files.
• sample_size: The number of samples to use (default is 256).

Other parameters such as learning rate (lr), epochs, and validation settings are also managed within these config files.

train_data:
  csv_path: [Replace with .csv Annotation File Path]
  video_folder: [Replace with Video Folder Path]
  sample_size: 256

AnimateDiff v3 (2023.12) utilizes Domain Adapter LoRA for flexible image model finetuning and implements SparseCtrl encoders (RGB image or scribble) to control animation content via condition maps.

Available Models:

• v3_adapter_sd_v15.ckpt: Domain Adapter (97.4 MB)
• v3_sd15_mm.ckpt: Motion Module (1.56 GB)
• v3_sd15_sparsectrl_scribble.ckpt: SparseCtrl Encoder for scribble conditions (1.86 GB)
• v3_sd15_sparsectrl_rgb.ckpt: SparseCtrl Encoder for RGB image conditions (1.85 GB)

Limitations & Best Practices:

• Small flickering may be noticeable.
• Visual quality for general Text-to-Video (T2V) may be limited due to lack of specific optimizations for community models.
• Style Alignment: For image animation or interpolation, it is recommended to use images generated by the same community model used for the animation.

AnimateDiff v2 (2023.09) features a motion module trained on larger resolutions and batch sizes for improved motion quality and diversity. It also supports MotionLoRA for eight basic camera movements.

Available Models:

• mm_sd_v15_v2.ckpt: Motion Module (1.7 GB)
• MotionLoRA Checkpoints (approx. 74-77 MB each):
  • v2_lora_ZoomIn.ckpt
  • v2_lora_ZoomOut.ckpt
  • v2_lora_PanLeft.ckpt
  • v2_lora_PanRight.ckpt
  • v2_lora_TiltUp.ckpt
  • v2_lora_TiltDown.ckpt
  • v2_lora_RollingClockwise.ckpt
  • v2_lora_RollingAnticlockwise.ckpt

The original version of AnimateDiff.

Available Models:

• mm_sd_v14.ckpt: Motion Module (1.6 GB)
• mm_sd_v15.ckpt: Motion Module (1.6 GB)