DOCSEARCHbyAlgolia
Submit Docs

FramePack

repository·main·Indexed 12 days ago

https://github.com/lllyasviel/framepack

A next-frame prediction neural network structure for progressive video generation. FramePack compresses input contexts to a constant length to enable long video generation (up to 1 minute) with low VRAM requirements, supporting NVIDIA RTX 30XX, 40XX, and 50XX series GPUs with a minimum of 6GB VRAM.

Tokens
1.6K
Snippets
3
Records
7
Agent score
47%

What's inside FramePack

  1. How the FramePack GUI works

    main

    The FramePack GUI is split into two main areas:

    • Left Panel: Used for uploading a reference image and entering a text prompt.
    • Right Panel: Displays the generated videos and latent previews.

    Key Behavior: Because FramePack uses next-frame-section prediction, videos are generated progressively. You will see a progress bar for each section and a latent preview for the upcoming section. If the generated video appears very short initially (e.g., only 1 second), this is expected behavior; the model will continue generating more sections to complete the requested length.

  2. Understanding the influence of TeaCache and Quantization

    main

    FramePack supports optimizations like teacache, sage-attention, bnb quant, and gguf. While these improve speed or memory efficiency, they are not strictly lossless and can influence the visual output.

    Best Practice Recommendation: Use teacache or other optimizations to quickly iterate and test ideas. Once you are satisfied with the prompt and settings, switch to the full diffusion process (turning off teacache and using standard precision) to generate the final high-quality result.

  3. Prompting Guidelines for FramePack

    main

    To achieve high-quality video generation, use short, motion-focused prompts. Focus on describing how the scene becomes dynamic rather than static details.

    Best Practices

    • Focus on Motion: Describe human activity, moving objects, or camera movements.
    • Prefer Large Motions: Larger, more dynamic motions (e.g., dancing, jumping, running) are preferred over subtle ones (e.g., standing, sitting).
    • Structure: Describe the subject, then the motion, then other details.
    • Conciseness: Brief, concise phrases generally work better than long descriptions.

    Example Prompts

    • The girl dances gracefully, with clear movements, full of charm.
    • The man dances powerfully, with clear movements, full of energy.
    • The young man writes intensely, flipping papers and adjusting his glasses with swift, focused movements.

  4. Install FramePack on Windows

    main

    FramePack provides a one-click package for Windows users containing CUDA 12.6 and PyTorch 2.6.

    1. Download the One-Click Package.
    2. Uncompress the downloaded file.
    3. Run update.bat to ensure you have the latest version and fixes.
    4. Run run.bat to start the software.

    Note: Models are downloaded automatically from HuggingFace upon first run; expect a download size of over 30GB.

    # Windows workflow
# 1. Uncompress package
# 2. Run update.bat
# 3. Run run.bat

  5. Hardware Requirements for FramePack

    main

    FramePack is a desktop software designed for next-frame prediction video diffusion. To run it effectively, ensure your hardware meets the following specifications:

    • GPU: NVIDIA RTX 30XX, 40XX, or 50XX series (must support fp16 and bf16). GTX 10XX/20XX series are not officially tested.
    • VRAM: Minimum 6GB of GPU memory is required. For example, generating a 60-second video (1800 frames at 30fps) using a 13B model requires at least 6GB VRAM.
    • OS: Linux or Windows.

    Performance Note: On an RTX 4090, generation speeds are approximately 2.5s/frame (unoptimized) or 1.5s/frame with teacache. Laptop GPUs (e.g., 3070ti, 3060) will be significantly slower (4x to 8x).

  6. Install FramePack on Linux

    main

    For Linux users, it is recommended to use an independent Python 3.10 environment.

    1. Install PyTorch with CUDA 12.6 support:
      pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu126
    2. Install other dependencies:
      pip install -r requirements.txt
    3. Launch the Gradio GUI:
      python demo_gradio.py

    The demo_gradio.py script supports standard Gradio flags such as --share, --port, and --server.

    Optional: Install Sage-Attention

    To improve performance on Linux, you can install sage-attention. Note that it may slightly influence results, so it is recommended to test it after verifying your setup without it.

    pip install sageattention==1.0.6
    # Linux installation steps
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu126
pip install -r requirements.txt
python demo_gradio.py

  7. Use ChatGPT to generate motion prompts

    main

    You can use the following ChatGPT system prompt to automate the generation of motion-focused prompts for FramePack. The goal is to transform an image into a single, concise motion description.

    System Prompt Template:

    You are an assistant that writes short, motion-focused prompts for animating images.

When the user sends an image, respond with a single, concise prompt describing visual motion (such as human activity, moving objects, or camera movements). Focus only on how the scene could come alive and become dynamic using brief phrases.

Larger and more dynamic motions (like dancing, jumping, running, etc.) are preferred over smaller or more subtle ones (like standing still, sitting, etc.).

Describe subject, then motion, then other things. For example: "The girl dances gracefully, with clear movements, full of charm."

If there is something that can dance (like a man, girl, robot, etc.), then prefer to describe it as dancing.

Stay in a loop: one image in, one motion prompt out. Do not explain, ask questions, or generate multiple options.