DOCSEARCHbyAlgolia
Submit Docs

DiTCtrl

repository·main·Indexed 43 days ago

https://github.com/tencentarc/ditctrl

A tuning-free approach for Multi-Modal Diffusion Transformers (MM-DiT) that enables coherent, long-duration video generation using multiple sequential prompts. It treats multi-prompt generation as a temporal video editing task with smooth transitions and supports tasks such as Word Swap and Reweight editing. Designed for use with CogVideoX-2B weights and T5 text encoders.

Tokens
1.6K
Snippets
6
Records
7
Agent score
6%

What's inside DiTCtrl

  1. Download CogVideoX-2B model weights

    main

    DiTCtrl requires CogVideoX-2B weights and a T5 text encoder.

    1. Download CogVideoX-2B weights

    Download the VAE and Transformer weights and organize them into the following structure:

    CogVideoX-2b-sat/
├── transformer
│   ├── 1000 (or 1)
│   │   └── mp_rank_00_model_states.pt
│   └── latest
└── vae
    └── 3d-vae.pt

    2. Download T5 Text Encoder

    Clone the CogVideoX-2b repository from HuggingFace or Modelscope. Move the text_encoder and tokenizer contents into a new directory named t5-v1_1-xxl to ensure a safetensor format that can be loaded without error.

    3. Final Directory Structure

    Your sat/ directory should look like this:

    sat/
├── CogVideoX-2b-sat/
│   ├── transformer
│   ├── CogVideoX-2b
│   ├── t5-v1_1-xxl
│   ├── vae
├── configs/
├── inference_case_configs/
├── run_multi_prompt.sh
├── run_single_prompt.sh
├── run_edit_video.sh 
├── sample_video.py
├── sample_video_edit.py
├── README.md
├── LICENSE
└── ...
    # Download CogVideoX-2B weights
cd sat
mkdir CogVideoX-2b-sat
cd CogVideoX-2b-sat

# Download VAE
wget https://cloud.tsinghua.edu.cn/f/fdba7608a49c463ba754/?dl=1
mv 'index.html?dl=1' vae.zip
unzip vae.zip

# Download Transformer
wget https://cloud.tsinghua.edu.cn/f/556a3e1329e74f1bac45/?dl=1
mv 'index.html?dl=1' transformer.zip
unzip transformer.zip

# Download T5
git clone https://huggingface.co/THUDM/CogVideoX-2b.git
mkdir t5-v1_1-xxl
mv CogVideoX-2b/text_encoder/* CogVideoX-2b/tokenizer/* t5-v1_1-xxl

  2. Generate prompts using LLM instructions

    main

    To get the best results for long video generation, use the instruction files provided in the prompts_gen_instruction folder when prompting an LLM:

    • For Single-prompts: Follow the instructions in the CogVideoX repository.
    • For Multi-prompts: Use one of the following:
      • presto.md: Focuses on realistic cinematographic sequences and natural camera movements (ideal for documentaries).
      • ditctrl.md: Focuses on creative scene transitions and imaginative scenarios (ideal for artistic/fantasy content).

  3. Setup the DiTCtrl environment

    main

    DiTCtrl requires CUDA 12 and is tested on single A100 or V100 GPUs. Follow these steps to create a Conda environment and install dependencies:

    1. Create and activate a new Conda environment with Python 3.10.
    2. Install PyTorch 2.4.1 with CUDA 12.1 support.
    3. Install project requirements via pip install -r requirements.txt.
    4. Install xformers specifically for the environment.

    Note: The environment is similar to CogVideo.

    cd DiTCtrl

conda create -n ditctrl python=3.10
conda activate ditctrl

pip install torch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1 --index-url https://download.pytorch.org/whl/cu121

pip install -r requirements.txt

conda install https://anaconda.org/xformers/xformers/0.0.28.post1/download/linux-64/xformers-0.0.28.post1-py310_cu12.1.0_pyt2.4.1.tar.bz2

  4. Run DiTCtrl generation tasks

    main

    Navigate to the sat directory to run the provided shell scripts for different generation modes:

    1. Longer Multi-Prompt Text-to-Video: Uses multiple sequential prompts for coherent long videos.
    2. Longer Single-Prompt Text-to-Video: Enhances consistency for single prompts by treating them as sequential identical prompts.
    3. Video Editing: Performs tasks like 'Word Swap' (using KV-sharing) or 'Reweight' (reweighting attention columns/rows).
    cd sat

# Multi-prompt generation
bash run_multi_prompt.sh

# Single-prompt generation
bash run_single_prompt.sh

# Video editing
bash run_edit_video.sh

  5. Configure custom inference via YAML

    main

    You can customize generation by providing a specific configuration file in the inference_case_configs/ folder. This file will overwrite the default settings in configs/inference.yaml.

    To run a custom configuration, use the --custom-config flag with sample_video.py.

    Example rose.yaml structure:

    args:
  is_run_isolated: False  # If True, generates isolated videos without DiTCtrl method
  seed: 42
  output_dir: outputs/multi_prompt_case/rose
  prompts:     # Sequential prompts for long video generation
    - "A gentle close shot of the same rose petal..."
    - "A steady medium shot of the rose..."
    - "A smooth full shot of the rose bush..."
    # Example command to run a custom config
inference_case_config="inference_case_configs/multi_prompts/rose.yaml"
run_cmd="python sample_video.py --base configs/cogvideox_2b.yaml configs/inference.yaml --custom-config $inference_case_config"
eval ${run_cmd}

  6. Troubleshoot T5 deserialization errors

    main

    If you encounter the error safetensors rust.SafetensorError: Error while deserializing header: HeaderTooLarge, it indicates the T5 model was not downloaded correctly. The t5-v1_1-xxl folder should be approximately 8.9GB.

    To fix this, use the HuggingFace mirror to download the model correctly:

    export HF_ENDPOINT=https://hf-mirror.com
huggingface-cli download THUDM/CogVideoX-2b --local-dir ./CogVideoX-2b