DOCSEARCHbyAlgolia
Submit Docs

Kohya_ss GUI

repository·master·Indexed 11 days ago

https://github.com/bmaltais/kohya_ss

A Gradio-based interface and CLI for training diffusion models, including LoRA, Dreambooth, and fine-tuning. Version 26.0.0 automates complex command-line arguments for underlying training scripts and supports local installation on Windows, Linux, and macOS, as well as cloud deployments via Docker, Runpod, and Google Colab.

Tokens
61.9K
Snippets
148
Records
253
Agent score
94%

What's inside Kohya_ss

  1. Overview of supported training methods

    master

    The repository supports several training modalities for fine-tuning models, including:

    • Model Fine-tuning: Using fine_tune.py.
    • DreamBooth: Using train_db.py.
    • LoRA: Using train_network.py.
    • Textual Inversion: Using train_textual_inversion.py (including XTI:P+).

    Training methods are categorized by how they handle data and captions:

    | Training Target | Script | DB / class+identifier | DB / Caption | Fine-tuning |
| :--- | :--- | :---: | :---: | :---: |
| Fine-tuning Model | `fine_tune.py` | x | x | o |
| DreamBooth Model | `train_db.py` | o | o | x |
| LoRA | `train_network.py` | o | o | o |
| Textual Inversion | `train_textual_inversion.py` | o | o | o |

  2. Overview of Fine-tuning in Kohya_ss

    master

    Fine-tuning in this project refers to training a model using images and captions. This process targets the Stable Diffusion U-Net using the Diffusers library. It is distinct from LoRA, Textual Inversion, or Hypernetworks, though Hypernetwork training is supported as a specific feature.

    Key capabilities include:

    • CLIP Output Modification: Option to use the second-to-last layer of the CLIP (Text Encoder) instead of the last layer for better prompt adherence (standard for SD 2.0).
    • Aspect Ratio Bucketing: Training on non-square resolutions (e.g., 256x1024) to reduce cropping and improve prompt-image relationship learning.
    • Token Length Extension: Expanding the maximum token length from 75 to 225 by splitting and concatenating CLIP calls.
    • Automated Captioning/Tagging: Support for BLIP (captioning), DeepDanbooru, and WD14Tagger (tagging).
    • Memory Optimization: Support for VAE output pre-caching, 8-bit AdamW, gradient checkpointing, and mixed precision (fp16/bf16).

  3. Overview of gen_img_diffusers.py

    master

    The gen_img_diffusers.py script is a Diffusers-based inference (image generation) tool designed for command-line use. It supports SD 1.x and 2.x models, LoRA, and ControlNet (v1.0 verified).

    Key Capabilities:

    • Supports txt2img, img2img, and inpainting.
    • Modes: Interactive mode, batch generation from a single prompt, or batch generation from a file.
    • Precision: Supports fp16, bf16 (for RTX 30 series), and fp32 (default).
    • Optimization: Supports xformers for faster, memory-efficient generation.
    • Advanced Features: CLIP skip, custom VAE loading, Highres. fix (custom implementation), LoRA (multiple simultaneous LoRAs with weight merging), and Attention Couple.

  4. Choose a training method (DreamBooth, Fine-tuning, LoRA, or Textual Inversion)

    master

    The library supports several training paradigms. Your choice depends on your training data and whether you want to use captions or regularization images:

    • DreamBooth (class + identifier): Best for training LoRA or Textual Inversion without preparing caption files. You associate a target with a specific identifier and class (e.g., shs dog). Note: This can make it harder to change attributes like clothing or background later.
    • DreamBooth (caption): Best if you can provide a .caption file for every image. This allows for better separation of the target from other elements (e.g., describing a character wearing 'white clothes').
    • Fine-tuning: Best for large datasets where you use a metadata JSON file instead of individual caption files. This method does not support regularization images.
    Training TargetScriptDB/class+idDB/captionFine-tuning
    Fine-tuning modelfine_tune.pyxxo
    DreamBooth modeltrain_db.pyoox
    LoRAtrain_network.pyooo
    Textual Inversiontrain_textual_inversion.pyooo

  5. What is DreamBooth and how does it work in Kohya_ss

    master

    DreamBooth is a technique used to fine-tune generative models (like Stable Diffusion) to learn a specific subject (e.g., a character or a specific art style) using a unique identifier (e.g., shs).

    Kohya's implementation of DreamBooth includes several enhancements over the original Diffusers script:

    • Memory optimization using 8-bit Adam optimizer and latent caching.
    • Memory optimization via xformers.
    • Support for arbitrary training resolutions (not just 512x512).
    • Support for image augmentation to improve quality.
    • Support for both Text Encoder + U-Net fine-tuning and U-Net only training.
    • Ability to read/write in Stable Diffusion format.
    • Aspect Ratio Bucketing.
    • Support for Stable Diffusion v2.0.

  6. Overview of ControlNet-LLLite

    master

    ControlNet-LLLite is a lightweight version of ControlNet (LoRA Like Lite) inspired by LoRA architecture. It is currently designed exclusively for SDXL.

    An LLLite module consists of a conditioning image embedding (which maps the conditioning image to the latent space) and a small network similar to a LoRA. These modules are added to the U-Net's Linear and Conv layers. Due to inference constraints, it currently only adds to CrossAttention (attn1's q/k/v and attn2's q).

  7. Configure sample image generation prompts

    master

    During training, you can use a prompt file to generate sample images. Lines starting with # are treated as comments. You can append specific generation options to the end of a prompt line using the following flags:

    • --n: Negative prompt (applies to everything until the next option).
    • --w: Width of the generated image.
    • --h: Height of the generated image.
    • --d: Seed of the generated image.
    • --l: CFG scale.
    • --s: Number of steps.

    Prompt weighting using ( ) and [ ] is supported.

    # prompt 1
masterpiece, best quality, (1girl), in white shirts, upper body, looking at viewer, simple background --n low quality, worst quality, bad anatomy, bad composition, poor, low effort --w 768 --h 768 --d 1 --l 7.5 --s 28

# prompt 2
masterpiece, best quality, 1boy, in business suit, standing at street, looking back --n (low quality, worst quality), bad anatomy, bad composition, poor, low effort --w 576 --h 832 --d 2 --l 5.5 --s 40

  8. Enable masked loss training

    master

    Masked loss allows you to calculate loss only for specific pixels. To enable this, specify the --masked_loss option in your training script.

    Dataset Requirements:

    • Use a ControlNet dataset to specify the mask.
    • Masks must be RGB images.
    • Mask logic: The pixel value 255 in the R channel is treated as the mask (loss calculated). A value of 0 is treated as non-mask.
    • Weighting: Pixel values between 0-255 are converted to a 0-1 range (e.g., a value of 128 results in half the loss weight).

  9. How LoRA structure works (Rank and Dimensions)

    master

    LoRA functions by adding a small, separate neural network to specific positions in the target model. This small network consists of three layers:

    • Input Layer: Matches the number of neurons in the target network's input.
    • Output Layer: Matches the number of neurons in the target network's output.
    • Middle Layer: The number of neurons in this layer is defined by the rank number (also referred to as the number of dimensions).

    This architecture allows for efficient training and distribution, as only the small LoRA weights need to be shared rather than the entire 2GB-5GB model.

  10. Important considerations for DreamBooth training

    master

    When training with DreamBooth in this repository, keep the following in mind:

    • Step Count: Due to memory optimization (splitting training and regularization images into different batches), the number of steps per iteration is half of the original script. To match the training intensity of the original Diffusers or XavierXiao scripts, double your max_train_steps.
    • Batch Size: DreamBooth trains the entire model (similar to full fine-tuning), so memory consumption is significantly higher than LoRA training. If you have sufficient VRAM, increasing the batch size in your .toml file can improve speed and accuracy.
    • Learning Rate: While Diffusers uses 5e-6, the Stable Diffusion version used here typically uses 1e-6.

  11. Configure Gradual Latent for Hires fix

    master

    Gradual Latent is a Hires fix technique that gradually increases the size of the latent during generation. It is available in gen_img.py, sdxl_gen_img.py, and gen_img_diffusers.py.

    Important Requirements:

    • You must use the euler_a sampler, as the sampler source code is modified specifically for this feature. Other samplers will not work.
    • It is most effective with SD 1.5; the effect is subtle with SDXL.

    Configuration Options:

    • --gradual_latent_timesteps (or --glt): The timestep at which to start increasing the latent size. Default is None (disabled). A starting value of 750 is recommended.
    • --gradual_latent_ratio (or --glr): The initial size of the latent. Default is 0.5 (half the default size).
    • --gradual_latent_ratio_step (or --gls): The increment amount for the latent size. Default is 0.125, resulting in a progression of 0.625, 0.75, 0.875, and 1.0.
    • --gradual_latent_ratio_every_n_steps (or --gle): The interval (in steps) at which the size increases. Default is 3.

  12. Understand LoRA types: LoRA-LierLa vs LoRA-C3Lier

    master

    The repository supports two custom LoRA types for training:

    1. LoRA-LierLa (read as "Liela"): Applies LoRA to Linear layers and Conv2d layers with a 1x1 Kernel. This is compatible with the standard AUTOMATIC1111 Web UI LoRA functionality.
    2. LoRA-C3Lier (read as "Seria"): Applies LoRA to Linear layers and Conv2d layers with a 3x3 Kernel. This may provide higher accuracy because it covers more layers, but requires the sd-webui-additional-networks extension to generate images in Web UI.

    Note: These implementations are incompatible with the cloneofsimo repository or the d8ahazard Dreambooth extension due to functional differences.