CogVideo & CogVideoX

VEnhancer is a unified framework designed to enhance videos generated by CogVideoX. It provides three main capabilities:

1. Spatial Super-resolution: Increasing the resolution of the video.
2. Temporal Super-resolution: Frame interpolation to increase frame rates.
3. Video Refinement: Reducing artifacts and improving visual quality.

Technical Architecture: VEnhancer utilizes a design similar to ControlNet. It copies the architecture and weights of the multi-frame encoder and middle block from a pre-trained video diffusion model to create a trainable conditional network. This network accepts low-resolution keyframes and noisy full-frame latents as inputs. Conditioning is provided via the time step t, the text prompt, the noise augmentation level σ, and the downscaling factor s.

To achieve better results during fine-tuning, follow these recommendations:

• Concept Training: Use videos with 25 or more frames when training new concepts or styles.
• Identifier Tokens: Use an identifier token (via --id_token) for better results, similar to Dreambooth training.
• LoRA Rank: Use a rank of 64 or higher.
• LoRA Alpha: Instead of the default lora_alpha = 1, set lora_alpha to be equal to the rank or rank // 2 for better performance.
• Data Caching: The system automatically encodes and caches videos to disk. If you modify your dataset after a training run, delete the latent directory under your video directory to ensure the cache is refreshed.

After downloading the weights, ensure your directory structure matches the following pattern for the inference engine to locate files correctly:

.
├── transformer
│   ├── 1000 (or 1)
│   │   └── mp_rank_00_model_states.pt
│   └── latest
└── vae
    └── 3d-vae.pt

To run CogVideoX on hardware with limited memory (e.g., NVIDIA T4 Colab), you can use quantization for the text encoder, transformer, and VAE modules.

Supported Tools

• PytorchAO and Optimum-quanto: Used to quantize modules to reduce memory requirements.
• TorchAO Quantization: Fully compatible with torch.compile, which can significantly improve inference speed.

Precision Recommendations

• CogVideoX-2B: Trained in FP16; use FP16 for inference.
• CogVideoX-5B: Trained in BF16; use BF16 for inference.
• NVIDIA H100 and newer: Use FP8 precision. This requires installing torch and torchao from source and is recommended for CUDA 12.4.

For LoRA models

Refer to ../inference/cli_demo.py for instructions on how to load and run inference with LoRA weights.

For SFT models

SFT training produces checkpoints that need to be merged into a single weight file. Use the zero_to_fp32.py script located in the checkpoint-* directory to integrate the weights.

To use the CogVLM2-Caption model for generating video captions (used for CogVideoX training data), install the necessary dependencies using pip:

pip install -r requirements.txt

Fine-tuned models cannot be merged. To use them, modify the run_cmd in inference.sh to point to the correct base and inference configs.

Example command structure:

run_cmd="$environs python sample_video.py --base configs/cogvideox_<model parameters>_lora.yaml configs/inference.yaml --seed 42"

Execution:

bash inference.sh

There are two primary paths for fine-tuning:

1. Diffusers-based LoRA: Use finetune/train_cogvideox_lora for fine-tuning CogVideoX models using the diffusers framework.
2. SAT-based: Use sat/README_zh.md for inference and fine-tuning using SAT weights. This is recommended for researchers looking to improve upon the CogVideoX architecture.
3. Low-cost Fine-tuning: The cogvideox-factory framework supports low-cost fine-tuning of the diffusers version and can run on a single RTX 4090 for CogVideoX-5B.

Datasets for fine-tuning must follow a specific directory structure where videos and their corresponding text labels are paired by filename:

.
├── labels
│   ├── 1.txt
│   ├── 2.txt
│   └── ...
└── videos
    ├── 1.mp4
    ├── 2.mp4
    └── ...

Requirements:

• Each .txt file must have the same name as its corresponding .mp4 file.
• The .txt file contains the label/prompt for that video.
• Maintain a one-to-one mapping between videos and labels.
• Style Fine-tuning: It is recommended to prepare at least 50 videos and labels with a similar style to improve fitting quality.