CogView4 Documentation

When running CogView4 (DIT models) with BF16 precision and batchsize=4, use the following memory requirements as a guide. It is recommended to have at least 32GB of RAM to prevent process termination.

Create an accelerate_ds.yaml configuration file to set up the deepspeed and accelerate environments. Below is a standard configuration using zero2 for a single machine with 8 processes:

compute_environment: LOCAL_MACHINE
debug: false
deepspeed_config:
  # deepspeed_hostfile: hostfile # If using multi-machine multi-card training, prepare the hostfile configuration
  gradient_accumulation_steps: 1
  gradient_clipping: 1.0
  offload_optimizer_device: cpu
  offload_param_device: cpu
  zero3_init_flag: false
  zero_stage: 2
num_machines: 1
num_processes: 8 # 8 processes in total, write 16 if using two machines
distributed_type: DEEPSPEED
downcast_bf16: 'no'
enable_cpu_affinity: false
gpu_ids: all
machine_rank: 0
main_training_function: main
mixed_precision: bf16
rdzv_backend: static
same_network: true
tpu_env: [ ]
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: false

The CogView4 series models are trained on long, descriptive synthetic image captions. To significantly improve generation quality, it is strongly recommended to rewrite your initial prompts using a Large Language Model (LLM) before generating images.

The repository provides a script inference/prompt_optimize.py to automate this refinement. Note that CogView4 and CogView3 require different few-shot examples for optimization, so you must specify the correct version using the --cogview_version flag.

cd inference
python prompt_optimize.py --api_key "Zhipu AI API Key" --prompt {your prompt} --base_url "https://open.bigmodel.cn/api/paas/v4" --model "glm-4-plus" --cogview_version "cogview4"

While CogView4 is trained on long descriptive prompts, it is strongly recommended to use a Large Language Model (LLM) to rewrite/polish your prompts before generation to significantly improve quality.

A script inference/prompt_optimize.py is provided for this purpose. Note that the few-shot prompting logic differs between CogView4 and CogView3, so ensure you specify the correct --cogview_version.

cd inference
python prompt_optimize.py --api_key "智谱AI API Key" --prompt {你的提示词} --base_url "https://open.bigmodel.cn/api/paas/v4" --model "glm-4-plus" --cogview_version "cogview4"

To begin training, clone the diffusers repository, install it in editable mode, and navigate to the CogView4 control examples directory:

git clone https://github.com/huggingface/diffusers.git
cd diffusers
pip install -e .
cd examples/cogview4-control

When running DIT models with BF16 precision and batchsize=4, you can manage VRAM usage using enable_model_cpu_offload and 4-bit Text Encoder quantization.

Recommended Hardware: A device with at least 32GB of RAM is recommended to prevent process termination.

You can use the open_pose_controlnet dataset or your own. If using a custom dataset, it must follow the open_pose_controlnet format or you must modify the dataloader accordingly.

Constraints:

• Resolution: All images will be resized to a fixed size; dynamic resolutions are not supported.
• Tokens: Dynamic-length tokens are not supported. Tokens in each batch will be padded to the maximum length.

You can train the CogView4-6B-Control model using the training code provided in the HuggingFace diffusers repository.

Model Link: CogView4-6B-Control Training Code: diffusers/examples/cogview4-control