DOCSEARCHbyAlgolia
Submit Docs

Hunyuan-DiT Documentation

repository·main·Indexed 26 days ago

https://github.com/tencent-hunyuan/hunyuandit

A multi-resolution Diffusion Transformer for high-fidelity text-to-image generation with specialized support for Chinese and English. The ecosystem includes model definitions, pre-trained weights, and training scripts for full-parameter and LoRA. It supports multiple inference modes including Diffusers, ComfyUI, Gradio, and Lite versions, with integrated support for ControlNet (Canny, Depth, Pose) and distilled base models for faster generation.

Tokens
24.3K
Snippets
63
Records
92
Agent score
88%

What's inside Hunyuan-DiT

  1. Overview of Hunyuan-DiT

    main

    Hunyuan-DiT is a multi-resolution Diffusion Transformer (DiT) designed for high-quality text-to-image generation with fine-grained understanding of both English and Chinese. It supports multi-turn text-to-image generation through multi-modal dialogue and features a bilingual DiT architecture.

    Key capabilities include:

    • Bilingual Support: Fine-grained understanding of Chinese and English prompts.
    • Multi-turn Generation: Ability to refine images through conversational context.
    • Extensibility: Support for ControlNet (Canny, Pose, Depth), IP-Adapter, and LoRA training.
    • Optimization: Available versions for acceleration via Distillation and TensorRT, as well as a lite version for 6GB GPU VRAM inference.

  2. Run HunyuanDiT Lite Inference

    main

    Use the lite/inference.py script to perform text-to-image generation. The script requires the model ID, a text prompt, the number of inference steps, and the guidance scale as positional arguments.

    cd HunyuanDiT

# Quick start
model_id=Tencent-Hunyuan/HunyuanDiT-v1.2-Diffusers-Distilled
prompt=一个宇航员在骑马
infer_steps=50
guidance_scale=6
python3 lite/inference.py ${model_id} ${prompt} ${infer_steps} ${guidance_scale}

  3. Create a Multireso Index V2 Dataset via Configuration

    main

    You can create a multi-resolution dataset by providing a configuration file to the idk tool. This process generates multiple resolution buckets based on a base resolution and either a step size or specific target aspect ratios.

    Configuration Fields

    Field NameTypeDescription
    srcRequiredPath(s) to the Base Index V2 file (single or multiple)
    base_sizeRequiredThe base resolution $(n, n)$ to start from. Recommended: 256, 512, or 1024
    reso_stepOptionalStep size for traversing resolutions. Use this OR target_ratios. Recommended: 16, 32, or 64
    target_ratiosOptionalA list of target aspect ratios (e.g., 1:1, 4:3, 3:4, 16:9, 9:16). Use this OR reso_step
    alignOptionalThe multiple to align the target resolution to when using target_ratios. Recommended: 16
    min_sizeOptionalMinimum resolution filter for samples. Recommended: 256, 512, or 1024
    md5_fileOptionalPath to a pre-calculated dictionary of image sizes in .pkl format (key: MD5, value: (h, w))
    src:
    - /HunYuanDiT/dataset/porcelain/jsons/a.json
    - /HunYuanDiT/dataset/porcelain/jsons/b.json
    - /HunYuanDiT/dataset/porcelain/jsons/c.json
base_size: 512
reso_step: 32
min_size: 512

  4. Train and use LoRA weights

    main

    HunyuanDiT supports LoRA training and inference.

    Training:

    • Use lora/train_lora_with_fa.sh if flash_attn is installed.
    • Use lora/train_lora.sh if flash_attn is NOT installed.

    Inference:

    • Use the --lora-ckpt flag in sample_t2i.py to apply trained weights.
    • You can download pre-trained LoRA weights (e.g., porcelain, jade) from Hugging Face.
    # Training for porcelain LoRA (with flash_attn)
PYTHONPATH=./ sh lora/train_lora_with_fa.sh --index-file dataset/porcelain/jsons/porcelain.json

# Training for porcelain LoRA (without flash_attn)
PYTHONPATH=./ sh lora/train_lora.sh --index-file dataset/porcelain/jsons/porcelain.json

# Inference using trained LORA weights
python sample_t2i.py --infer-mode fa --prompt "青花瓷风格，一只小狗"  --no-enhance --lora-ckpt log_EXP/001-lora_porcelain_ema_rank64/checkpoints/0001000.pt

# Quick start with downloaded Hugging Face LoRA weights
cd HunyuanDiT
huggingface-cli download Tencent-Hunyuan/HYDiT-LoRA --local-dir ./ckpts/t2i/lora
python sample_t2i.py --infer-mode fa --prompt "青花瓷风格，一只猫在追蝴蝶"  --no-enhance --load-key ema --lora-ckpt ./ckpts/t2i/lora/porcelain

  5. Set up Hunyuan DiT 1.2 ControlNet in ComfyUI

    main

    To use ControlNet features with Hunyuan DiT 1.2 in ComfyUI:

    1. Download ControlNet weights: Download the desired ControlNet model weight files.
    2. Install weights: Place them in the ComfyUI/models/controlnet/hunyuandit directory.
    3. Load the workflow: Drag and drop a ControlNet example image (like hunyuan_dit_1.2_controlnet_canny_example.png) into ComfyUI to load the canny ControlNet workflow.

    Available ControlNet Models:

    Trained by @tencent:

    • Pose: pytorch_model_pose_distill.pt
    • Depth: pytorch_model_depth_distill.pt
    • Canny: pytorch_model_canny_distill.pt

    Trained by @TTPlanetPig:

    • Inpaint ControlNet
    • Tile ControlNet
    • Lineart ControlNet

    Required Custom Nodes for @TTPlanetPig models:

    • Comfyui_TTP_CN_Preprocessor
    • Comfyui_TTP_Toolset

  7. Download HunyuanDiT ControlNet weights

    main

    To use ControlNet, you need to download both the ControlNet weights and the distilled base model weights. It is recommended to use distilled weights as the base model for ControlNet inference because the provided pretrained weights were trained on them.

    Run the following commands from the HunyuanDiT directory:

    cd HunyuanDiT
# Download ControlNet weights
huggingface-cli download Tencent-Hunyuan/HYDiT-ControlNet-v1.2 --local-dir ./ckpts/t2i/controlnet

# Download distilled base model weights
huggingface-cli download Tencent-Hunyuan/Distillation-v1.2 ./pytorch_model_distill.pt --local-dir ./ckpts/t2i/model

  8. Download HunyuanDiT ControlNet v1.1 weights

    main

    To use HunyuanDiT ControlNet v1.1, you must download both the ControlNet weights and the distilled base model weights. It is highly recommended to use the distilled weights as the base model for ControlNet inference, as the provided pretrained weights were trained on them.

    cd HunyuanDiT
# Download ControlNet weights
huggingface-cli download Tencent-Hunyuan/HYDiT-ControlNet-v1.1 --local-dir ./HunyuanDiT-v1.1/t2i/controlnet

# Download distilled base model weights
huggingface-cli download Tencent-Hunyuan/Distillation-v1.1 ./pytorch_model_distill.pt --local-dir ./HunyuanDiT-v1.1/t2i/model

  10. Repeat samples during dataset creation

    main

    You can repeat samples using three methods. If multiple conditions apply, the highest repeat count is used.

    1. Source Repeat: Add repeat: n directly to a source path (ensure a colon follows the path).
    2. Keyword Repeat: Use the repeater section with arrow_file_keyword to repeat based on file name patterns.
    3. MD5 Repeat: Use the repeater section with an md5 field to repeat based on specific MD5s in a .txt, .json, or .pkl file.
    # 1. Source Repeat
source:
    - /HunYuanDiT/dataset/porcelain/arrows/*.arrow:
        repeat: 10

# 2. Keyword Repeat
repeater:
    arrow_file_keyword:
        - repeat: 8
          keyword:
            - Lolita anime style
            - Minimalist style

# 3. MD5 Repeat
repeater:
    md5:
        - name: goodcase1
          path: /HunYuanDiT/dataset/porcelain/md5_repeat_1.json
          type: dict
          plus: 3