DOCSEARCHbyAlgolia
Submit Docs

Wan2.1 Video Generative Models

repository·main·Indexed 12 days ago

https://github.com/wan-video/wan2.1

An open-source suite of large-scale video foundation models supporting Text-to-Video (T2V), Image-to-Video (I2V), First-Last-Frame-to-Video (FLF2V), and video editing (VACE). Optimized for consumer-grade GPUs, Wan2.1 features a 3D causal variational autoencoder (Wan-VAE) and supports both 1.3B and 14B parameter versions. It integrates with Hugging Face diffusers, ComfyUI, and Gradio, and is licensed under Apache 2.0.

Tokens
8.4K
Snippets
23
Records
32
Agent score
95%

What's inside Wan2.1

  1. Overview of Wan2.1 Video Generative Models

    main

    Wan2.1 is an open suite of large-scale video foundation models designed for high-performance video generation. Key capabilities include:

    • Multiple Tasks: Supports Text-to-Video (T2V), Image-to-Video (I2V), Video Editing, Text-to-Image, and Video-to-Audio.
    • Consumer-grade GPU Support: The T2V-1.3B model is optimized for consumer hardware, requiring approximately 8.19 GB of VRAM. On an RTX 4090, it can generate a 5-second 480P video in about 4 minutes (without quantization).
    • Visual Text Generation: Capable of generating robust Chinese and English text within videos.
    • Powerful Video VAE: The Wan-VAE efficiently encodes and decodes 1080P videos of any length while preserving temporal information.

  2. Understand the Wan2.1 model architectures

    main

    Wan2.1 utilizes a Diffusion Transformer (DiT) paradigm with a Flow Matching framework. The architecture is supported in two primary scales:

    ModelDimensionInput DimOutput DimFF DimensionFreq DimensionHeadsLayers
    1.3B1536161689602561230
    14B51201616138242564040

    Key architectural components:

    • Wan-VAE: A 3D causal variational autoencoder designed for video. It supports encoding/decoding unlimited-length 1080P videos without losing temporal information.
    • T5 Encoder: Used to encode multilingual text input.
    • Modulation: An MLP with a Linear layer and a SiLU layer processes input time embeddings to predict six modulation parameters, shared across all transformer blocks.

  3. Available Wan2.1 Model Tasks and Integrations

    main

    Wan2.1 provides several specialized models and task-specific implementations. As of the latest updates, the following tasks are supported:

    Supported Tasks

    • Text-to-Video (T2V): Available in 1.3B and 14B parameter versions.
    • Image-to-Video (I2V): Available in 14B parameter version.
    • First-Last-Frame-to-Video (FLF2V): Generates video conditioned on a first and last frame (14B model).
    • VACE: An all-in-one model for video creation and editing.

    Ecosystem Integrations

    • Diffusers: T2V and I2V models are integrated into the Hugging Face diffusers library via WanPipeline and WanImageToVideoPipeline.
    • ComfyUI: Native support is available for Wan2.1 models.
    • Gradio: Inference code includes Gradio demos for most models.

  4. Install Wan2.1

    main

    To install Wan2.1, clone the repository and install the required dependencies. Ensure you have torch >= 2.4.0 installed.

    git clone https://github.com/Wan-Video/Wan2.1.git
cd Wan2.1
pip install -r requirements.txt

  5. Run Text-to-Video (T2V) Generation

    main

    Generate videos from text prompts using either the 1.3B or 14B models.

    Single-GPU Inference

    Use the generate.py script. For 14B models, you can specify resolution with --size (e.g., 1280*720).

    If you encounter Out-of-Memory (OOM) issues, use --offload_model True and --t5_cpu to reduce GPU memory usage.

    Example (1.3B with memory optimization):

    python generate.py --task t2v-1.3B --size 832*480 --ckpt_dir ./Wan2.1-T2V-1.3B --offload_model True --t5_cpu --sample_shift 8 --sample_guide_scale 6 --prompt "Two anthropomorphic cats in comfy boxing gear and bright gloves fight intensely on a spotlighted stage."

    Multi-GPU Inference (FSDP + xDiT USP)

    Accelerate inference using FSDP and xDiT. You can use Ulysses or Ring strategies.

    • Ulysses Strategy: Set --ulysses_size $GPU_NUMS. Note: num_heads must be divisible by ulysses_size.
    • Ring Strategy: Set --ring_size $GPU_NUMS. Note: sequence length must be divisible by ring_size.

    Example (8 GPUs with Ulysses):

    pip install "xfuser>=0.4.1"
torchrun --nproc_per_node=8 generate.py --task t2v-14B --size 1280*720 --ckpt_dir ./Wan2.1-T2V-14B --dit_fsdp --t5_fsdp --ulysses_size 8 --prompt "Two anthropomorphic cats in comfy boxing gear and bright gloves fight intensely on a spotlighted stage."
    python generate.py  --task t2v-14B --size 1280*720 --ckpt_dir ./Wan2.1-T2V-14B --prompt "Two anthropomorphic cats in comfy boxing gear and bright gloves fight intensely on a spotlighted stage."

  6. Run VACE (Video Editing and Generation)

    main

    VACE supports video generation and editing using text prompts and optional video, mask, or image inputs.

    Single-GPU Inference

    python generate.py --task vace-1.3B --size 832*480 --ckpt_dir ./Wan2.1-VACE-1.3B --src_ref_images examples/girl.png,examples/snake.png --prompt "[Chinese Prompt]"

    Multi-GPU Inference

    torchrun --nproc_per_node=8 generate.py --task vace-14B --size 1280*720 --ckpt_dir ./Wan2.1-VACE-14B --dit_fsdp --t5_fsdp --ulysses_size 8 --src_ref_images examples/girl.png,examples/snake.png --prompt "[Chinese Prompt]"
    python generate.py --task vace-1.3B --size 832*480 --ckpt_dir ./Wan2.1-VACE-1.3B --src_ref_images examples/girl.png,examples/snake.png --prompt "在一个欢乐而充满节日气氛的场景中..."

  7. Run First-Last-Frame-to-Video (FLF2V) Generation

    main

    Generate video transitions between a first and last frame. Currently only 720P is supported.

    Single-GPU Inference

    python generate.py --task flf2v-14B --size 1280*720 --ckpt_dir ./Wan2.1-FLF2V-14B-720P --first_frame examples/flf2v_input_first_frame.png --last_frame examples/flf2v_input_last_frame.png --prompt "CG animation style..."
    python generate.py --task flf2v-14B --size 1280*720 --ckpt_dir ./Wan2.1-FLF2V-14B-720P --first_frame examples/flf2v_input_first_frame.png --last_frame examples/flf2v_input_last_frame.png --prompt "CG animation style, a small blue bird takes off from the ground."

  8. Download Wan2.1 Models

    main

    Models can be downloaded from Huggingface or ModelScope. Available models include T2V (Text-to-Video), I2V (Image-to-Video), FLF2V (First-Last-Frame-to-Video), VACE, and T2I (Text-to-Image) variants in 1.3B and 14B sizes.

    Using huggingface-cli:

    pip install "huggingface_hub[cli]"
huggingface-cli download Wan-AI/Wan2.1-T2V-14B --local-dir ./Wan2.1-T2V-14B

    Using modelscope-cli:

    pip install modelscope
modelscope download Wan-AI/Wan2.1-T2V-14B --local_dir ./Wan2.1-T2V-14B

  11. Optimize Wan2.1 performance on different GPU configurations

    main

    When running Wan2.1 models, you can adjust specific CLI flags to manage memory and computational efficiency depending on your hardware:

    • For the 1.3B model on 8 GPUs: Use --ring_size 8 and --ulysses_size 1 to distribute the workload.
    • For the 14B model on a single GPU: Use --offload_model True to manage memory constraints.
    • For the 1.3B model on a single consumer GPU (e.g., RTX 4090): Use --offload_model True --t5_cpu to offload the T5 encoder to the CPU and save VRAM.

    Note: Prompt extension is disabled by default in these efficiency tests; to enable it, use the --use_prompt_extend flag. Also, be aware that T2V-14B (Text-to-Video) is slower than I2V-14B (Image-to-Video) because T2V samples 50 steps while I2V uses 40 steps.

    # Example configuration for 1.3B model on a single 4090
python generate.py --model_type 1.3B --offload_model True --t5_cpu

  12. Run Text-to-Image (T2I) Generation

    main

    Wan2.1 is a unified model that can also generate high-quality images.

    Single-GPU Inference

    python generate.py --task t2i-14B --size 1024*1024 --ckpt_dir ./Wan2.1-T2V-14B --prompt '一个朴素端庄的美人'

    Multi-GPU Inference

    torchrun --nproc_per_node=8 generate.py --dit_fsdp --t5_fsdp --ulysses_size 8 --base_seed 0 --frame_num 1 --task t2i-14B --size 1024*1024 --prompt '一个朴素端庄的美人' --ckpt_dir ./Wan2.1-T2V-14B
    python generate.py --task t2i-14B --size 1024*1024 --ckpt_dir ./Wan2.1-T2V-14B  --prompt '一个朴素端庄的美人'