DOCSEARCHbyAlgolia
Submit Docs

Wan2.2: Large-Scale Video Generative Models

repository·main·Indexed 12 days ago

https://github.com/wan-video/wan2.2

A suite of advanced video generative models featuring a Mixture-of-Experts (MoE) architecture. Wan2.2 supports high-quality text-to-video (T2V), image-to-video (I2V), and audio-driven speech-to-video (S2V) generation. Key models include the 14B MoE series for high-fidelity output and the TI2V-5B model optimized for consumer-grade hardware like the NVIDIA RTX 4090. It also includes Wan-Animate for character animation and replacement.

Tokens
10K
Snippets
27
Records
38
Agent score
97%

What's inside Wan2.2

  1. Overview of Wan2.2 Video Generative Models

    main

    Wan2.2 is a suite of advanced large-scale video generative models featuring a Mixture-of-Experts (MoE) architecture. Key capabilities include:

    • MoE Architecture: Uses specialized expert models for different denoising stages to increase capacity without increasing computational cost.
    • Cinematic Aesthetics: Trained on curated data with detailed labels for lighting, composition, and color for precise style control.
    • High-Definition Hybrid TI2V: A 5B model using a 16×16×4 compression ratio VAE, supporting Text-to-Video (T2V) and Image-to-Video (I2V) at 720P resolution (24fps). It is optimized to run on consumer-grade hardware like the NVIDIA RTX 4090.
    • Specialized Models: Includes Wan2.2-Animate-14B for character animation/replacement and Wan2.2-S2V-14B for audio-driven (Speech-to-Video) generation.

  2. Wan2.2-TI2V-5B High-Compression Video Generation

    main

    The TI2V-5B (Text-to-Video/Image-to-Video) model is a dense 5B parameter model designed for high efficiency and high-definition output.

    Key Features

    • Unified Framework: Natively supports both Text-to-Video (T2V) and Image-to-Video (I2V) tasks.
    • High Compression: Uses a Wan2.2-VAE with a $T \times H \times W$ compression ratio of $4 \times 16 \times 16$ (64x total). With an additional patchification layer, the ratio reaches $4 \times 32 \times 32$.
    • Performance: Can generate a 5-second 720P video in under 9 minutes on a single consumer-grade GPU.

  3. Understand Wan-animate generation modes

    main

    Wan-animate provides two distinct generation modes, each with different preprocessing requirements and behaviors:

    1. animation mode: Focuses on animating a character. It is highly recommended to enable pose retargeting if the body proportions of the reference character and the driving character differ significantly. This mode generates src_face.mp4 and src_pose.mp4 in the save_path.

    2. replacement mode: Focuses on replacing a character within a scene. Pose retargeting is DISABLED by default to preserve spatial interactions with the environment. This mode generates src_face.mp4, src_pose.mp4, src_bg.mp4, and src_mask.mp4 in the save_path.

    Warning for Replacement Mode: The built-in mask extraction is designed for single-person videos only. Multi-person videos may result in incorrect pose tracking or failure.

  4. How the Wan2.2 Mixture-of-Experts (MoE) Architecture works

    main

    The A14B model series uses a Mixture-of-Experts (MoE) architecture to increase total parameters while maintaining low inference costs.

    • Total Parameters: ~27B
    • Active Parameters per step: ~14B

    Expert Design

    The model switches between two specialized experts based on the Signal-to-Noise Ratio (SNR) during the denoising process:

    1. High-Noise Expert: Activated during the early stages (high noise/low SNR) to focus on the overall layout.
    2. Low-Noise Expert: Activated during the later stages (low noise/high SNR) to refine fine video details.

    The switch occurs at a threshold step $t_{moe}$ corresponding to half of the minimum SNR ($SNR_{min}$).

  5. Run Wan2.2: Installation Requirements and Tips

    main

    When setting up Wan2.2, ensure your environment meets the following requirements and follow these installation tips to avoid common errors:

    Requirements

    • PyTorch: Ensure torch >= 2.4.0 is installed.

    Installation Tips

    • Flash Attention: If the installation of flash_attn fails, install all other required packages first, and attempt to install flash_attn as the final step.
    • Speech-to-Video (S2V): If you intend to use CosyVoice for speech synthesis in Speech-to-Video generation tasks, you must additionally install the requirements specified in requirements_s2v.txt.

  6. Install Wan2.2 and dependencies

    main

    To set up the Wan2.2 environment, clone the repository and install the required Python packages.

    Important Notes:

    • Ensure torch >= 2.4.0 is installed.
    • If flash_attn installation fails, install all other packages first and install flash_attn last.
    • For Speech-to-Video generation using CosyVoice, you must additionally install requirements_s2v.txt.
    # Clone the repo
git clone https://github.com/Wan-Video/Wan2.2.git
cd Wan2.2

# Install standard dependencies
pip install -r requirements.txt

# Install additional requirements for Speech-to-Video (CosyVoice)
pip install -r requirements_s2v.txt

  8. Configure Wan2.2 for Multi-GPU and Single-GPU Inference

    main

    When running Wan2.2 models, you can optimize performance and memory usage using specific CLI flags.

    Multi-GPU Configuration

    For the 14B model, use Ulysses and FSDP to distribute the workload:

    • --ulysses_size 4/8
    • --dit_fsdp
    • --t5_fsdp

    For the 5B model, use:

    • --ulysses_size 4/8
    • --offload_model True
    • --convert_model_dtype
    • --t5_cpu

    Single-GPU Configuration

    To run on a single GPU, use model offloading and dtype conversion to manage memory:

    • 14B Model: --offload_model True --convert_model_dtype
    • 5B Model: --offload_model True --convert_model_dtype --t5_cpu

    Note: --convert_model_dtype converts model parameter types to the value specified in config.param_dtype.

    # Example Multi-GPU 14B setup
python run_inference.py --model_type 14B --ulysses_size 8 --dit_fsdp --t5_fsdp

# Example Single-GPU 5B setup
python run_inference.py --model_type 5B --offload_model True --convert_model_dtype --t5_cpu

  9. Run Text-Image-to-Video (TI2V) generation

    main

    Generate video using both text and an image via the Wan2.2-TI2V-5B model. Supports 720P.

    Resolution Note: For TI2V, 720P resolution must be specified as 1280*704 or 704*1280.

    Hardware Requirements

    • Can run on a GPU with at least 24GB VRAM (e.g., RTX 4090).
    • For 80GB+ VRAM, remove --offload_model True, --convert_model_dtype, and --t5_cpu to increase speed.

    Usage

    • Text-to-Video: Omit the --image parameter.
    • Image-to-Video: Include the --image parameter.
    # Single-GPU TI2V (Text + Image)
python generate.py --task ti2v-5B --size 1280*704 --ckpt_dir ./Wan2.2-TI2V-5B --offload_model True --convert_model_dtype --t5_cpu --image examples/i2v_input.JPG --prompt "Summer beach vacation style..."

  10. Set up preprocessing model checkpoints

    main

    The preprocessing pipeline requires specific models for pose detection, mask extraction, and image editing. Ensure your checkpoint directory follows this structure:

    /path/to/your/ckpt_path/
├── det/
│   └── yolov10m.onnx
├── pose2d/
│   └── vitpose_h_wholebody.onnx
├── sam2/
│   └── sam2_hiera_large.pt
└── FLUX.1-Kontext-dev/
    /path/to/your/ckpt_path/
├── det/
│   └── yolov10m.onnx
├── pose2d/
│   └── vitpose_h_wholebody.onnx
├── sam2/
│   └── sam2_hiera_large.pt
└── FLUX.1-Kontext-dev/

  11. Run Wan-Animate (Animation and Replacement)

    main

    Wan-Animate uses a video and a character image to generate new video in two modes:

    1. Animation Mode: The character image mimics the motion in the input video.
    2. Replacement Mode: The character image replaces the person in the input video.

    Workflow

    1. Preprocessing: You MUST preprocess input videos into materials using preprocess_data.py before running inference.
    2. Inference: Run via generate.py or the Diffusers pipeline.

    Diffusers Pipeline Usage

    If using the WanAnimatePipeline from diffusers:

    • For Animation: Provide image, pose_video, and face_video.
    • For Replacement: Provide image, pose_video, face_video, background_video, and mask_video.

    Warning: Do not use LoRA models trained on Wan2.2 with Wan-Animate, as weight changes may cause unexpected behavior.

    # Diffusers Animation Example
from diffusers import WanAnimatePipeline

pip install diffusers

pipe = WanAnimatePipeline.from_pretrained("Wan-AI/Wan2.2-Animate-14B-Diffusers", torch_dtype=torch.bfloat16)
pipe.to("cuda:0")

animate_video = pipe(
    image=image,
    pose_video=pose_video,
    face_video=face_video,
    prompt="People in the video are doing actions.",
    mode="animate",
    segment_frame_length=77,
    prev_segment_conditioning_frames=1,
    guidance_scale=1.0,
    num_inference_steps=20,
).frames[0]

  12. Run Text-to-Video (T2V) generation

    main

    Generate video from text using the Wan2.2-T2V-A14B model. Supports 480P and 720P.

    Single-GPU Inference

    Requires at least 80GB VRAM. Use --offload_model True, --convert_model_dtype, and --t5_cpu to mitigate OOM (Out-of-Memory) issues.

    Multi-GPU Inference

    Uses PyTorch FSDP and DeepSpeed Ulysses for acceleration.

    Prompt Extension

    To improve video quality, you can extend prompts using:

    1. Dashscope API: Requires DASH_API_KEY. Use --use_prompt_extend --prompt_extend_method 'dashscope'.
    2. Local Model: Uses Qwen models (e.g., Qwen/Qwen2.5-7B-Instruct). Use --use_prompt_extend --prompt_extend_method 'local_qwen'.
    # Single-GPU T2V
python generate.py --task t2v-A14B --size 1280*720 --ckpt_dir ./Wan2.2-T2V-A14B --offload_model True --convert_model_dtype --prompt "Two anthropomorphic cats in comfy boxing gear and bright gloves fight intensely on a spotlighted stage."

# Multi-GPU T2V with FSDP + Ulysses
torchrun --nproc_per_node=8 generate.py --task t2v-A14B --size 1280*720 --ckpt_dir ./Wan2.2-T2V-A14B --dit_fsdp --t5_fsdp --ulysses_size 8 --prompt "Two anthropomorphic cats in comfy boxing gear and bright gloves fight intensely on a spotlighted stage."