DOCSEARCHbyAlgolia
Submit Docs

LongCat-Video

repository·main·Indexed 26 days ago

https://github.com/meituan-longcat/longcat-video

A 13.6B parameter foundational video generation model supporting Text-to-Video, Image-to-Video, and Video-Continuation tasks. It features a coarse-to-fine generation strategy and Block Sparse Attention for efficient 720p 30fps video production. The repository also includes LongCat-Video-Avatar 1.5, an open-source framework for audio-driven human video generation utilizing Whisper-Large for improved lip sync and step distillation for accelerated inference.

Tokens
2.8K
Snippets
9
Records
14
Agent score
42%

What's inside LongCat-Video

  1. Overview of LongCat-Video

    main

    LongCat-Video is a foundational video generation model with 13.6B parameters. It is designed for high-quality, efficient long video generation and supports multiple tasks within a single framework, including:

    • Text-to-Video: Generating video from text prompts.
    • Image-to-Video: Generating video from static images.
    • Video-Continuation: Extending existing videos, enabling the production of minutes-long content without quality degradation or color drifting.

    The model uses a coarse-to-fine generation strategy and Block Sparse Attention to generate $720p$, $30fps$ videos efficiently. It was trained using multi-reward Group Relative Policy Optimization (GRPO).

  2. LongCat-Video-Avatar 1.5 Features

    main

    LongCat-Video-Avatar 1.5 is an upgraded open-source framework for audio-driven human video generation. Key improvements over previous versions include:

    • Improved Lip Sync: Replaces Wav2Vec2 with Whisper-Large for higher accuracy.
    • Stability: Achieves production-ready physical rationality and temporal stability for long videos.
    • Generalization: Supports stylized domains such as anime, animals, and complex real-world conditions.
    • Audio Support: Supports both single-stream and multi-stream audio inputs.
    • Inference Speed: Accelerated to 8 steps via step distillation.

  4. Usage Considerations for LongCat-Video

    main

    When deploying LongCat-Video, developers should be aware of the following:

    • Evaluation Limitations: The model has not been comprehensively evaluated for every possible downstream application.
    • LLM Characteristics: Be mindful of known large language model limitations, such as performance variations across different languages.
    • Safety and Compliance: Carefully assess accuracy, safety, and fairness before deployment in sensitive or high-risk scenarios. Developers are responsible for complying with all applicable laws regarding data protection, privacy, and content safety.
    • Licensing: The model is released under the MIT License.

  5. Run LongCat-Video-Avatar-1.5

    main

    Generate audio-driven videos using the Avatar 1.5 model. This version uses the Whisper-large-v3 audio encoder and supports distillation for faster inference.

    Key CLI Arguments for Avatar 1.5:

    • --model_type avatar-v1.5: Specifies the 1.5 model.
    • --use_distill: Required for 1.5 to enable distillation sampling (faster inference).
    • --use_int8: Enables INT8 quantization to reduce VRAM usage (only for 1.5).
    • --stage_1: Set to at2v (Audio-Text-to-Video) or ai2v (Audio-Image-to-Video).
    • --input_json: Path to the input configuration JSON.
    • --num_segments: Number of segments for video continuation.
    • --ref_img_index: Index for reference image (0-24 for consistency, 30 to reduce repeated actions).
    • --mask_frame_range: Frame range for masking (default 3).

    Examples:

    Single-Audio (Audio-Text-to-Video):

    torchrun --nproc_per_node=2 run_demo_avatar_single_audio_to_video.py --context_parallel_size=2 --checkpoint_dir=./weights/LongCat-Video-Avatar-1.5 --stage_1=at2v --input_json=assets/avatar/single_example_1.json --use_distill --model_type avatar-v1.5 --use_int8

    Single-Audio (Audio-Image-to-Video):

    torchrun --nproc_per_node=2 run_demo_avatar_single_audio_to_video.py --context_parallel_size=2 --checkpoint_dir=./weights/LongCat-Video-Avatar-1.5 --stage_1=ai2v --input_json=assets/avatar/single_example_1.json --use_distill --model_type avatar-v1.5 --use_int8

    Multi-Audio (Audio-Image-to-Video):

    torchrun --nproc_per_node=2 run_demo_avatar_multi_audio_to_video.py --context_parallel_size=2 --checkpoint_dir=./weights/LongCat-Video-Avatar-1.5 --input_json=assets/avatar/multi_example_1.json --use_distill --model_type avatar-v1.5 --use_int8
    torchrun --nproc_per_node=2 run_demo_avatar_single_audio_to_video.py --context_parallel_size=2 --checkpoint_dir=./weights/LongCat-Video-Avatar-1.5 --stage_1=at2v --input_json=assets/avatar/single_example_1.json --use_distill --model_type avatar-v1.5 --use_int8

  6. Install LongCat-Video

    main

    Follow these steps to set up the environment for LongCat-Video:

    1. Clone the repository:

      git clone --single-branch --branch main https://github.com/meituan-longcat/LongCat-Video
cd LongCat-Video

    2. Create and activate a Conda environment:

      conda create -n longcat-video python=3.10
conda activate longcat-video

    3. Install PyTorch (ensure CUDA version matches your system):

      pip install torch==2.6.0+cu124 torchvision==0.21.0+cu124 torchaudio==2.6.0 --index-url https://download.pytorch.org/whl/cu124

    4. Install FlashAttention-2 dependencies:

      pip install ninja psutil packaging
pip install flash_attn==2.7.4.post1

    5. Install remaining requirements:

      pip install -r requirements.txt

    6. Install Avatar-specific requirements:

      conda install -c conda-forge librosa ffmpeg
pip install -r requirements_avatar.txt
    git clone --single-branch --branch main https://github.com/meituan-longcat/LongCat-Video
cd LongCat-Video

conda create -n longcat-video python=3.10
conda activate longcat-video

pip install torch==2.6.0+cu124 torchvision==0.21.0+cu124 torchaudio==2.6.0 --index-url https://download.pytorch.org/whl/cu124

pip install ninja 
pip install psutil 
pip install packaging 
pip install flash_attn==2.7.4.post1

pip install -r requirements.txt

conda install -c conda-forge librosa
conda install -c conda-forge ffmpeg
pip install -r requirements_avatar.txt

  7. Download LongCat-Video Models

    main

    Use huggingface-cli to download the necessary model weights into the ./weights/ directory.

    Required models:

    • meituan-longcat/LongCat-Video: Foundational video generation.
    • meituan-longcat/LongCat-Video-Avatar: Single/multi-character audio-driven video generation.
    • meituan-longcat/LongCat-Video-Avatar-1.5: Upgraded avatar model with Whisper-large-v3 and fast inference.
    pip install "huggingface_hub[cli]"
huggingface-cli download meituan-longcat/LongCat-Video --local-dir ./weights/LongCat-Video
huggingface-cli download meituan-longcat/LongCat-Video-Avatar --local-dir ./weights/LongCat-Video-Avatar
huggingface-cli download meituan-longcat/LongCat-Video-Avatar-1.5 --local-dir ./weights/LongCat-Video-Avatar-1.5

  8. Run Interactive Video Generation

    main

    Execute interactive video generation using torchrun.

    Single-GPU:

    torchrun run_demo_interactive_video.py --checkpoint_dir=./weights/LongCat-Video --enable_compile

    Multi-GPU:

    torchrun --nproc_per_node=2 run_demo_interactive_video.py --context_parallel_size=2 --checkpoint_dir=./weights/LongCat-Video --enable_compile
    torchrun run_demo_interactive_video.py --checkpoint_dir=./weights/LongCat-Video --enable_compile

  9. Run Text-to-Video Generation

    main

    Execute text-to-video inference using torchrun. Use --enable_compile for optimized performance.

    Single-GPU:

    torchrun run_demo_text_to_video.py --checkpoint_dir=./weights/LongCat-Video --enable_compile

    Multi-GPU (requires --context_parallel_size):

    torchrun --nproc_per_node=2 run_demo_text_to_video.py --context_parallel_size=2 --checkpoint_dir=./weights/LongCat-Video --enable_compile
    torchrun run_demo_text_to_video.py --checkpoint_dir=./weights/LongCat-Video --enable_compile

  10. Run Long-Video Generation

    main

    Generate long-duration videos using torchrun.

    Single-GPU:

    torchrun run_demo_long_video.py --checkpoint_dir=./weights/LongCat-Video --enable_compile

    Multi-GPU:

    torchrun --nproc_per_node=2 run_demo_long_video.py --context_parallel_size=2 --checkpoint_dir=./weights/LongCat-Video --enable_compile
    torchrun run_demo_long_video.py --checkpoint_dir=./weights/LongCat-Video --enable_compile

  11. Run Image-to-Video Generation

    main

    Execute image-to-video inference using torchrun.

    Single-GPU:

    torchrun run_demo_image_to_video.py --checkpoint_dir=./weights/LongCat-Video --enable_compile

    Multi-GPU:

    torchrun --nproc_per_node=2 run_demo_image_to_video.py --context_parallel_size=2 --checkpoint_dir=./weights/LongCat-Video --enable_compile
    torchrun run_demo_image_to_video.py --checkpoint_dir=./weights/LongCat-Video --enable_compile

  12. Run Video-Continuation

    main

    Extend existing video sequences using torchrun.

    Single-GPU:

    torchrun run_demo_video_continuation.py --checkpoint_dir=./weights/LongCat-Video --enable_compile

    Multi-GPU:

    torchrun --nproc_per_node=2 run_demo_video_continuation.py --context_parallel_size=2 --checkpoint_dir=./weights/LongCat-Video --enable_compile
    torchrun run_demo_video_continuation.py --checkpoint_dir=./weights/LongCat-Video --enable_compile