DOCSEARCHbyAlgolia
Submit Docs

Musubi Tuner

repository·main·Indexed 23 days ago

https://github.com/kohya-ss/musubi-tuner

A collection of scripts for training LoRA (Low-Rank Adaptation) models for high-end architectures, including HunyuanVideo, Wan2.1/2.2, and FLUX.1. It features optimized training options such as H2D-only block swap for improved throughput and support for various attention modes including torch, xformers, and SageAttention.

Tokens
75K
Snippets
128
Records
333
Agent score
83%

What's inside musubi-tuner

  1. Overview of HiDream-O1-Image support

    main

    HiDream-O1-Image support in Musubi Tuner is experimental and follows a three-step workflow:

    1. Cache pixel patch tokens.
    2. Cache prompt token IDs.
    3. Train or run inference.

    Training and inference require a single model-weight argument: --dit, which points to a HiDream-O1 single checkpoint (.safetensors) or a compatible model weights directory.

    Recommended base models from Comfy-Org:

    • hidream_o1_image_bf16.safetensors
    • hidream_o1_image_dev_bf16.safetensors

    Tokenizer, processor, and config assets are automatically loaded from official HiDream repositories based on the --model_type (full or dev).

  2. Overview of Z-Image architecture

    main
    Z-Image is an experimental model architecture within the Musubi Tuner framework that supports text-to-image generation. Many configuration options for pre-caching, training, and inference are shared with HunyuanVideo. You can view all available options by running the command with the --help flag.

  3. Overview of Ideogram 4 in Musubi Tuner

    main

    Ideogram 4 is a text-to-image model supported by Musubi Tuner. It consists of a 34-layer single-stream transformer (DiT), a Qwen3-VL 8B Text Encoder, and a Flux2 KL-VAE.

    Key Technical Details:

    • Quantization: The DiT is distributed only in quantized forms (FP8 and NVFP4). Musubi Tuner loads the DiT from a pre-quantized FP8 checkpoint and keeps it frozen in FP8. Weights are dequantized on-the-fly to the compute dtype, and LoRA modules run in the compute dtype. This is the standard operating mode.
    • Asymmetric CFG: Inference utilizes separate conditional and unconditional DiT models.
    • Layout: This adapter targets the Comfy-Org single-file component layout.
    • License: The model license is non-commercial. Ensure you have accepted the terms before use.

  4. Overview of Musubi Tuner

    main

    Musubi Tuner is an unofficial command-line tool designed for LoRA training of various large-scale models, including:

    • HunyuanVideo
    • Wan2.1/2.2
    • FramePack
    • FLUX.1 Kontext
    • FLUX.2 dev/klein
    • Qwen-Image
    • Z-Image

    Note: This project is under active development and is not affiliated with the official repositories of the models mentioned.

  5. Overview of Wan 2.1/2.2 support in Musubi Tuner

    main

    Musubi Tuner provides unofficial training and inference scripts for Wan2.1 and Wan2.2.

    Key features include:

    • Memory Efficiency: Supports fp8 and block swapping. This allows inference of 720x1280x81 frame videos with 24GB VRAM, and training with 720x1280 images with 24GB VRAM.
    • Attention Support: Inference works without Flash Attention (using PyTorch's scaled dot product attention). It supports xformers (training and inference) and Sage attention (inference only).
    • Wan2.2 Support: Supports the Wan2.2 model architecture specifically for 14B models (experimental).

    Note: fp8_scaled models are not supported for either Wan2.1 or Wan2.2, even if the --fp8_scaled flag is used.

  6. Overview of Kandinsky 5 support

    main

    Musubi Tuner provides unofficial, experimental support for Kandinsky 5 training and inference. Key features include:

    • fp8 support and memory reduction via block swap.
    • Inference without Flash Attention (utilizes PyTorch's scaled dot product attention).
    • LoRA training for both Text-to-Video (T2V) and Image-to-Video (I2V/Pro) models.

  7. Overview of Qwen-Image architectures

    main

    Musubi Tuner supports several Qwen-Image architectures for different generation tasks:

    • Qwen-Image: Standard text-to-image generation.
    • Qwen-Image-Edit (including Edit-2509 and Edit-2511): Supports image editing using control images. While the official version supports up to 3 control images, Musubi Tuner allows specifying more (though operation is only confirmed up to 3). Control images can have different sizes during training and inference.
    • Qwen-Image-Layered: Supports image layer segmentation.

    Note: These features are currently experimental. Many training and inference options are shared with HunyuanVideo.

  8. Overview of Krea 2 (K2) Architecture

    main

    Krea 2 (K2) is an experimental text-to-image generation architecture within Musubi Tuner based on a single-stream MMDiT. It utilizes Qwen3-VL-4B-Instruct as the text encoder and the Qwen-Image VAE as the autoencoder.

    Model Variants

    • RAW Model: The full-step, CFG-based MMDiT transformer model. This is the model you should use for training.
    • Turbo Model: A distilled, few-step, CFG-free version. This is recommended for inference and optional sample image generation during training.

    Recommended Workflow: Train on the RAW model and run inference on the Turbo model.

  9. Overview of FramePack architecture in Musubi Tuner

    main

    FramePack is an experimental video generation architecture developed by lllyasviel, integrated into Musubi Tuner.

    Key Characteristics:

    • I2V Only: Supports Image-to-Video (I2V) generation; Text-to-Video (T2V) is not supported.
    • Architecture: Uses a unique DiT model and requires an additional Image Encoder (SigLIP). It shares the same VAE and Text Encoders as HunyuanVideo but uses the original FramePack method for utilization.
    • VRAM Efficiency: Due to its progressive generation nature, it can have significantly lower VRAM usage for long videos compared to other architectures.
    • Scripts: Uses specific caching and training scripts prefixed with fpack_*.py.

    For one-frame inference and training, refer to the framepack_1f.md guide. Many latent pre-caching, training, and inference options are shared with HunyuanVideo.

  10. What is LoRA Post-Hoc EMA merging?

    main

    LoRA Post-Hoc EMA (Exponential Moving Average) merging is a technique used to combine multiple LoRA checkpoint files into a single, more stable model. It works by sorting checkpoints by modification time (oldest to newest) and iteratively merging them using a decay rate (beta).

    Key features:

    • Temporal ordering: Automatically sorts files by modification time.
    • Configurable decay rates: Supports a single beta value or linear interpolation between two beta values (--beta and --beta2).
    • Metadata preservation: Uses metadata from the last checkpoint and updates the model hash.
    • Dtype preservation: Maintains the original data types of tensors (though calculations are performed in float32 for stability).
    • Non-float handling: Non-float tensors (e.g., long, int, bool) are not merged; the values from the first checkpoint are used.
    beta = 0.95
checkpoints = [checkpoint1, checkpoint2, checkpoint3]
merged_weights = checkpoints[0]
for checkpoint in checkpoints[1:]:
    merged_weights = beta * merged_weights + (1 - beta) * checkpoint

  11. Project Folder Structure

    main

    Musubi Tuner uses a fixed directory structure within the specified project folder to simplify management. The following files and folders are automatically managed by the GUI:

    + training : Folder for training images (images + .txt caption files). Created automatically.
+ cache : Folder for latent and text encoder pre-caches. Created during preprocessing.
+ models : Folder for trained LoRA models. Created at the start of training.
    + sample : Subfolder for sample images generated during training.
+ dataset_config.toml : Dataset configuration file. Created during dataset setup.
+ musubi_project.toml : Project settings file (stores model, resolution, etc.).

    Note on Training Data: Users must place images (*.jpg, *.png, etc.) and their corresponding caption files (same basename with .txt extension) into the training folder. The GUI will validate the existence of these files during the dataset configuration step.

    + training : 学習用画像を配置するフォルダ、GUIスクリプトが自動作成します。
+ cache : latentとtext encoderの事前キャッシュを配置するフォルダ（キャッシュ処理で自動的に作成されます）
+ models : 学習結果のLoRAモデルが配置されるフォルダ（学習開始で自動的に作成されます）
    + sample : Musubi Tunerが学習中のサンプル生成画像を出力するフォルダ
+ dataset_config.toml : データセット設定ファイル、GUIスクリプトが自動作成します。
+ musubi_project.toml : GUIのプロジェクト設定（選択したモデルや解像度など）を保存するファイル。

  12. Use H2D-only block swap for optimized training

    main

    The --block_swap_h2d_only option enables an optimized block swap mode for LoRA (LoHa/LoKr) training across all architectures.

    In frozen-base training, the base weights on the CPU and GPU are identical, making the standard device-to-host (D2H) copy redundant overhead. H2D-only keeps a permanent master copy on the CPU and only performs host-to-device (H2D) transfers, removing the D2H transfer entirely. This improves throughput, especially when using --fp8_base or --fp8_scaled.

    Requirements & Tuning:

    • Requires --gradient_checkpointing to be enabled.
    • Use --block_swap_ring_size to tune the number of GPU ring buffers used for streaming (default is 2; setting it to 1 minimizes VRAM usage).