sd-scripts

ControlNet-LLLite for Anima is a lightweight, LoRA-like conditional control module for the Anima DiT (MiniTrainDIT) architecture. It uses the v2 architecture, which features:

• A deep conditioning1 trunk for a wider receptive field.
• FiLM (γ, β) modulation inside each LLLite module (zero-initialized for identity start).
• Per-module depth embedding (zero-init bias) for layer-specificity.
• Atomic target_layers specifiers, including mlp_fc1_pre for MLP block injection.
• Optional ASPP (Atrous Spatial Pyramid Pooling) tail via --lllite_use_aspp.

Status: Experimental. Currently supports image generation only (T=1).

Unsupported Features: The following will cause the training script to assert/fail if enabled:

• --blocks_to_swap
• --cpu_offload_checkpointing
• --unsloth_offload_checkpointing
• --deepspeed
• --fused_backward_pass

In addition to standard LoRA, sd-scripts supports LoHa (Low-rank Hadamard Product) and LoKr (Low-rank Kronecker Product) as alternative parameter-efficient fine-tuning methods based on the LyCORIS project.

• LoHa: Represents weight updates as a Hadamard (element-wise) product of two low-rank matrices.
• LoKr: Represents weight updates as a Kronecker product with optional low-rank decomposition.

Both methods target Linear and Conv2d layers.

• Conv2d 1x1 layers: Treated similarly to Linear layers.
• Conv2d 3x3+ layers: Supports optional Tucker decomposition or flat (kernel-flattened) mode.

Note: This feature is experimental.

LECO (Low-rank adaptation for Erasing COncepts) is a technique for training LoRA models to modify or erase concepts from diffusion models without an image dataset. It uses only text prompts to train against the model's own noise predictions.

Capabilities:

• Concept erasing: Remove specific styles or concepts (e.g., "van gogh").
• Concept enhancing: Strengthen specific attributes (e.g., "detailed").
• Slider LoRA: Create bidirectional control (e.g., a slider between "short hair" and "long hair").

Key Differences from Standard LoRA:

• Data: No images required; uses a Prompt TOML instead of a Dataset TOML.
• Target: Trains the U-Net only (not Text Encoder).
• Units: Training is measured in steps only (not epochs).
• Saving: Saves per-step only using --save_every_n_steps.

The sd-scripts repository supports several model training methods, each with different data preparation requirements:

• Fine-tuning: Best for large datasets. Uses metadata files (JSON) containing captions and tags. Does not support regularization images.
• DreamBooth (class + identifier): Associates a target with a specific unique word (identifier). Good for learning specific characters without manual captions. Supports regularization images.
• DreamBooth (caption): Uses pre-written caption files for each image to allow better separation of elements (e.g., clothing, background). Supports regularization images.
• LoRA: Supports both class + identifier and caption methods.
• Textual Inversion: Supports both class + identifier and caption methods.

The repository supports two specific types of LoRA training:

1. LoRA-LierLa: Applies LoRA to Linear layers and 1x1 Conv2d kernels.
2. LoRA-C3Lier: Applies LoRA to Linear layers and 3x3 Conv2d kernels. This type covers more layers and may offer higher precision.

Note: LoRA-LierLa is compatible with the AUTOMATIC1111 Web UI. For LoRA-C3Lier, you must use the sd-webui-additional-networks extension in the Web UI.

Validation loss is a metric used to monitor the training process and assess how well a model generalizes to unseen data. It is essential for detecting overfitting.

In sd-scripts, the validation process is designed to be deterministic. This means the script uses the same random seed for noise generation and timestep selection during every validation run. Consequently, any fluctuations in the validation loss metric reflect actual changes in the model's weights rather than random noise in the validation procedure itself. The script logs the average loss across all validation steps as a single metric.

The gen_img.py script is a command-line tool for inference (image generation) supporting SD 1.x, 2.x, and SDXL models. It supports txt2img, img2img, and inpainting, as well as LoRA, ControlNet, and various samplers.

Key features include:

• Support for fp16 and bf16 (for RTX 30+ GPUs).
• Support for xformers and SDPA (Scaled Dot-Product Attention).
• Prompt weighting (using () and []) and negative prompts (using --n).
• Highres. fix and upscaling support.
• Support for multiple LoRAs and ControlNet (v1.0/v1.1/LLLite).

The repository supports two training methods: Fine-tuning and LoRA (Low-Rank Adaptation).

Fine-tuning

Retrains all (or most) of the weights of a pre-trained model.

• Pros: Higher expressive power; suitable for major style or concept changes.
• Cons: High VRAM/compute cost; large file sizes (full model size); high risk of overfitting.
• Scripts: *_train.py (e.g., sdxl_train.py, sd3_train.py, flux_train.py).

LoRA tuning

Freezes model weights and trains a small additional "adapter" network.

• Pros: Low VRAM/compute cost; fast training; small file sizes (MBs); resistant to overfitting; supports combining multiple LoRAs.
• Cons: May not achieve as significant a change as fine-tuning.
• Scripts: *_train_network.py (e.g., sdxl_train_network.py, sd3_train_network.py, flux_train_network.py).

The lumina_train_network.py script is used to train LoRA models for Lumina Image 2.0. It shares many features with train_network.py, including sample image generation via --sample_prompts and detailed optimizer settings.

Upon completion, the LoRA model file (e.g., my_lumina_lora.safetensors) is saved to the directory specified by --output_dir. These models can be used in inference environments like ComfyUI (with appropriate nodes) or via the provided lumina_minimal_inference.py script.

The repository provides the following capabilities:

• LoRA training
• Fine-tuning (DreamBooth) for models other than HunyuanImage-2.1
• Textual Inversion training (SD/SDXL)
• Inpainting model training (SD1.5 and SDXL)
• Image generation
• Utilities for model conversion, tagging, and LoRA merging

hunyuan_image_train_network.py is a specialized script for training LoRA models on the HunyuanImage-2.1 DiT (Diffusion Transformer) architecture.

Key Differences from train_network.py:

• Architecture: Uses a Transformer-based (DiT) structure instead of standard Stable Diffusion.
• Text Encoders: Requires two encoders: Qwen2.5-VL and byT5.
• Arguments: Requires specific arguments for the DiT, Qwen2.5-VL, byT5, and VAE files.
• Incompatible Options: Arguments like --v2, --clip_skip, and --max_token_length are not supported.
• Specific Features: Includes unique parameters for flow matching.