LyCORIS Documentation

LoKr (LoRA for Kronecker product) decomposes weight updates using the Kronecker product ($\otimes$). The weight update is expressed as $\Delta W = W_1 \otimes (Wa_2 \cdot Wb_2)$, where $W_1$ acts as a weight scale for a standard LoRA-decomposed matrix $W_2$.

Key Properties:

• Multiplicative Rank: The rank of the resulting $\Delta W$ is multiplicative: $rank(\Delta W) \le min(a, b) \times r$, where $r$ is the LoRA rank.
• Parameter Efficiency: It can reduce the number of parameters to the order of the square root of the matrix dimensions. For a matrix of size $m \times n$, if the factor is set to -1, the number of parameters is approximately $\sqrt{mn} + r \times (\sqrt{m} + \sqrt{n})$.

LyCORIS allows for expanding LoRA-style fine-tuning from transformer layers to the whole model, including convolutional layers (LoCon).

Key differences:

• Standard LoRA: Typically targets transformer layers only.
• LoCon: Expands the adaptation to convolutional layers, allowing the model to learn more complex features. While fine-tuning the whole model can capture more information, it also carries a higher risk of heavy overfitting.

For detailed algorithm comparison guidelines, refer to the Guidelines in this repository.

LoHa (LoRA for Hadamard product) uses the Hadamard product ($\odot$) to decompose weight updates: $\Delta W = B \odot A$. This allows the rank of $\Delta W$ to be up to $rank(B) \times rank(A)$.

Memory Optimization: Because computing the gradient for $\Delta W = (B_1 \cdot B_2) \odot (A_1 \cdot A_2)$ normally requires PyTorch to cache both $B$ and $A$ (doubling memory usage), LyCORIS implements a custom backward pass. This custom backward reconstructs $B$ and $A$ only when needed during the backward pass, significantly reducing memory consumption during training.

LoCon (LoRA for Convolution) extends LoRA to convolutional layers. While standard LoRA for convolution can be implemented as a sequence of a $1\times1$ convolution and a standard convolution, LoCon supports a more efficient method called Tucker decomposition when use_tucker=True is enabled.

In Tucker decomposition, the weight shape $[out, in, kw, kh]$ is decomposed into multiple layers:

1. A $1\times1$ convolution: Conv(in, dim, (1, 1))
2. A kernel-sized convolution: Conv(dim, dim, (kw, kh), stride, padding)
3. A final $1\times1$ convolution: Conv(dim, out, (1, 1))

This allows for more efficient parameter usage compared to simple unfolding.

For sd-webui version 1.5.0 and later, LyCORIS models are natively supported by the built-in LoRA system.

1. Place your models in either models/Lora or models/LyCORIS.
2. Trigger the model using the standard syntax: <lora:filename:multiplier>.

For older versions (< 1.5.0) or sd-webui-forge, use the a1111-sd-webui-lycoris extension.

You can train LyCORIS modules using the lycoris.kohya module within the kohya-ss/sd-scripts framework.

Using command line arguments: Pass the --network_module lycoris.kohya flag along with necessary dimensions and algorithm arguments.

Using TOML configuration files: Provide a config file and a dataset config file to the training script.

Example command line usage:

accelerate launch train_network.py \
  --network_module lycoris.kohya \
  --network_dim "DIM_FOR_LINEAR" --network_alpha "ALPHA_FOR_LINEAR"\
  --network_args "conv_dim=DIM_FOR_CONV" "conv_alpha=ALPHA_FOR_CONV" \
  "dropout=DROPOUT_RATE" "algo=locon" \

To use LyCORIS for training or as a standalone wrapper, install the package in your corresponding virtual environment using one of the following methods:

Via pip:

pip install lycoris-lora

From source:

git clone https://github.com/KohakuBlueleaf/LyCORIS
cd LyCORIS
pip install .

LyCORIS provides several presets to quickly configure which layers are trained in the UNet and CLIP. These are useful for common training scenarios:

• preset=full: The default preset. Trains all layers in the UNet and CLIP.
• preset=full-lin: Similar to full, but skips convolutional layers.
• preset=attn-mlp: The "kohya preset". Trains all transformer blocks.
• preset=attn-only: Only attention layers are trained.
• preset=unet-transformer-only: Trains only the UNet transformer blocks (similar to kohya_ss/sd_scripts with disabled Text Encoder).
• preset=unet-convblock-only: Only ResBlock, UpSample, and DownSample layers are trained.

Use batch_bundle_convert.py to "zip" (bundle) or "unzip" (separate) network checkpoints and their corresponding embeddings.

Conversion Directions

• To Bundle (--to_bundle): Groups network checkpoints and embeddings together based on matching step counts (format: {name}-{step_count}.{ext}). The resulting file is named {network_name}-bundle-{step_count}.{network_ext}.
• From Bundle (--from_bundle): Separates a single bundle file into one network checkpoint and multiple embedding files, appending the step count to the embedding filenames.

# Example: Bundle network checkpoints and embeddings
python batch_bundle_convert.py \
--network_path /path/to/sd-webui-ssd/models/Lora  \
--emb_path /path/to/ckpts \
--dst_dir /path/to/sd-webui-ssd/models/Lora/bundle \
--to_bundle --verbose 2