DOCSEARCHbyAlgolia
Submit Docs

DynamiCrafter Documentation

repository·main·Indexed 25 days ago

https://github.com/doubiiu/dynamicrafter

A video diffusion prior model for animating open-domain still images using text prompts. It supports generative frame interpolation, looping video generation, and image-to-video generation across multiple resolutions (256, 512, and 1024). The repository includes instructions for installation via Anaconda, CLI and Gradio-based inference, and fine-tuning procedures for both image-to-video and frame interpolation tasks.

Tokens
1.3K
Snippets
5
Records
8
Agent score
34%

What's inside DynamiCrafter

  1. Fine-tune DynamiCrafter for Image-to-Video Generation

    main

    To fine-tune the model for Image-to-Video generation, follow these requirements and steps:

    Prerequisites

    • Dataset: Download the WebVid Dataset. The .csv files must contain page_dir, videoid, and name columns.
    • Models: Place pretrained models in the appropriate checkpoints/ directories.
    • Strategy: The process uses DDPShardedStrategy by default; ensure pytorch_lightning is installed and supports it.

    Setup

    1. Update <YOUR_SAVE_ROOT_DIR> in the training run script (e.g., training_1024_v1.0/run.sh).
    2. Verify all paths in the configuration file (e.g., training_1024_v1.0/config.yaml), specifically:
      • model:pretrained_checkpoint
      • data:data_dir
      • data:meta_path

    Execution

    Run the training script corresponding to your target resolution (1024 or 512). All checkpoints, Tensorboard records, and logs will be saved to your specified <YOUR_SAVE_ROOT_DIR>.

  2. Fine-tune DynamiCrafter for Frame Interpolation

    main

    To fine-tune the model specifically for frame interpolation:

    1. Download the DynamiCrafter512_interp pretrained model and place model.ckpt in checkpoints/dynamicrafter_512_interp_v1/model.ckpt.
    2. Follow the same dataset and configuration setup steps used for Image-to-Video fine-tuning.
    3. Execute the interpolation-specific training script.
    sh configs/training_512_v1.0/run_interp.sh

  3. Generate Frame Interpolation or Looping Videos via Command Line

    main

    To use DynamiCrafter for frame interpolation or looping video generation:

    1. Download the DynamiCrafter512_interp pretrained model.
    2. Place the model.ckpt in checkpoints/dynamicrafter_512_interp_v1/model.ckpt.
    3. Run the scripts/run_application.sh script with the desired application argument.
    # Generate frame interpolation
sh scripts/run_application.sh interp

# Looping video generation
sh scripts/run_application.sh loop

  4. Install DynamiCrafter via Anaconda

    main

    To set up the DynamiCrafter environment, it is recommended to use Anaconda. Create a new environment with Python 3.8.5, activate it, and install the required dependencies using the provided requirements.txt file.

    conda create -n dynamicrafter python=3.8.5
conda activate dynamicrafter
pip install -r requirements.txt

  5. Perform Image-to-Video Generation via Command Line

    main

    To generate video from images using the CLI, follow these steps:

    1. Download pretrained models from Hugging Face.
    2. Place the model.ckpt file in the directory corresponding to your required resolution: checkpoints/dynamicrafter_[1024|512|256]_v1/model.ckpt.
    3. Execute the appropriate script based on your hardware configuration.

    Use scripts/run.sh for single GPU inference and scripts/run_mp.sh for multi-GPU parallel inference.

    # Run on a single GPU (replace 1024 with 512 or 256 as needed):
sh scripts/run.sh 1024

# Run on multiple GPUs for parallel inference:
sh scripts/run_mp.sh 1024

  6. Run Local Gradio Demos

    main

    You can launch a web-based interface using Gradio. Ensure pretrained models are placed in the correct checkpoints/ directories as described in the inference guides.

    For Image-to-Video Generation, use gradio_app.py and specify the resolution with the --res flag.

    For Frame Interpolation and Looping, use gradio_app_interp_and_loop.py.

  7. Reference: Model Hardware Requirements and Checkpoints

    main

    Summary of available models, their resolutions, and minimum GPU memory requirements.

    |Model|Resolution|GPU Mem. |Checkpoint|
|:---------|:---------|:--------|:--------|
|DynamiCrafter1024|576x1024|10GB |[Hugging Face](https://huggingface.co/Kijai/DynamiCrafter_pruned/blob/main/dynamicrafter_1024_v1_bf16.safetensors)|
|DynamiCrafter512_interp|320x512|8GB |[Hugging Face](https://huggingface.co/Kijai/DynamiCrafter_pruned/blob/main/dynamicrafter_512_interp_v1_bf16.safetensors)|

  8. Available DynamiCrafter Models

    main

    DynamiCrafter offers several model checkpoints optimized for different resolutions and tasks. Note that inference time and memory usage are based on an A100 GPU using 50 DDIM steps.

    ModelResolutionGPU Mem. & Inference Time (A100, ddim 50steps)Checkpoint
    DynamiCrafter1024576x102418.3GB & 75s (perframe_ae=True)Hugging Face
    DynamiCrafter512320x51212.8GB & 20s (perframe_ae=True)Hugging Face
    DynamiCrafter256256x25611.9GB & 10s (perframe_ae=False)Hugging Face
    DynamiCrafter512_interp320x51212.8GB & 20s (perframe_ae=True)Hugging Face