PEFT (Parameter-Efficient Fine-Tuning)

🤗 PEFT (Parameter-Efficient Fine-Tuning) is a library designed to adapt large pretrained models to downstream tasks without the prohibitive cost of full fine-tuning.

Instead of updating all model parameters, PEFT methods only fine-tune a small number of additional parameters. This approach:

• Significantly decreases computational and storage costs.
• Enables training and storing large language models (LLMs) and other large-scale models on consumer hardware.
• Achieves performance comparable to fully fine-tuned models.

PEFT acts as a framework for various model adaptation methods (such as modifying weights, wrapping layers, or manipulating KV-caches) and serves as a reference implementation for many fine-tuning techniques.

The KappaTune experiment compares different fine-tuning strategies to evaluate task adaptation versus catastrophic forgetting. It measures performance on a downstream task (e.g., gsm8k) against a general-knowledge control dataset (e.g., WikiText).

Strategies compared:

• Baseline: No adaptation.
• LoRA_Global: Classic LoRA applied to common projections (q_proj, k_proj, o_proj, v_proj, gate_proj, up_proj, down_proj).
• KappaTune_LoRA: Uses the KappaTuneSelector with relative selection (e.g., top_p=0.2).

When to use KappaTune: Use KappaTune when catastrophic forgetting is a concern. If your fine-tuning data is closely aligned with the model's pretraining distribution (e.g., math data for certain models), unrestricted LoRA might be better as it reinforces pre-training, whereas KappaTune is designed to preserve original knowledge while adapting.

BEFT (Bias-Efficient Fine-Tuning) is a parameter-efficient fine-tuning algorithm that specifically fine-tunes only the added bias terms of value projections ($b_v$) in pretrained transformer models.

When to use BEFT:

• Low-data regimes: BEFT is designed to excel when you have limited training data. It requires significantly fewer parameters than LoRA while maintaining competitive or superior performance.

When to avoid BEFT:

• High-data regimes: In scenarios with abundant data, BEFT may show limited effectiveness compared to LoRA or full-parameter fine-tuning. In these cases, consider using LoRA instead.

LoKr is a LoRA-variant method that approximates a large weight matrix using two low-rank matrices combined via a Kronecker product. It can optionally use a third low-rank matrix for better fine-tuning control.

Key characteristics:

• Rank Preservation: By expressing the weight update matrix as a decomposition of a Kronecker product (creating a block matrix), LoKr can preserve the rank of the original weight matrix.
• Efficiency: The Kronecker product can be vectorized by stacking matrix columns, which avoids the need to fully reconstruct the weight update matrix $\Delta W$, speeding up the process.
• Parameterization: The size of the smaller matrices is determined by the rank parameter r.

WaveFT is a parameter-efficient fine-tuning (PEFT) method that learns sparse updates in the wavelet domain of residual matrices.

Unlike LoRA, which relies on discrete low-rank matrices, WaveFT learns a sparse set of coefficients in a transformed space. These coefficients are mapped back to the weight domain using the Inverse Discrete Wavelet Transform (IDWT). This approach allows for high-rank updates without adding inference overhead and provides finer control over the number of trainable parameters, making it highly effective in extreme low-parameter regimes.

Constraint:

• Currently, only nn.Linear layers are supported.

Block-Diagonal LoRA (BD-LoRA) is a variant of LoRA where some LoRA factors are constrained to be block-diagonal.

Key Benefits:

• Faster Serving: Eliminates communication overheads when running inference across multiple GPUs.
• Performance: Maintains finetuning performance comparable to vanilla LoRA.

Requirements:

• To realize inference speed-ups, you must use an inference engine compatible with BD-LoRA. Currently, experimental support is available via a specific vLLM fork.

AdaMSS (Adaptive Multi-Subspace Approach for Parameter-Efficient Fine-Tuning) is a PEFT method that decomposes weight matrices using SVD and clusters the decomposed space into multiple trainable subspaces. Each subspace learns independent low-rank updates while the original weights remain frozen.

Key features include:

• Subspace Segmentation: Leverages multiple smaller subspaces instead of a single large subspace.
• Adaptive Subspace Allocation (ASA): Dynamically prunes less important subspaces during training based on gradient information to reduce trainable parameters.
• Efficiency: Aims for more compact expressiveness and finer tuning compared to single-subspace methods like LoRA or PiSSA.

Constraints:

• Only nn.Linear layers are supported.
• Requires scikit-learn for the KMeans clustering step.

Activated LoRA (aLoRA) is a specialized adapter designed to selectively activate its weights only after a specific invocation sequence. This mechanism ensures that hidden states match the base model prior to the invocation point.

Key Benefit: It allows for the reuse of base model Key-Value (KV) caches for all tokens preceding the invocation. This enables significantly faster real-world inference (e.g., in engines like vLLM) when switching between base model generation and adapter-based generation.

MiSS (Matrix Shard Sharing) is a Parameter-Efficient Fine-Tuning (PEFT) method designed to balance model performance and computational efficiency. Unlike LoRA, which uses low-rank decomposition, MiSS decomposes weight matrices into multiple fragment matrices and utilizes a shared, trainable common fragment. This shard-sharing mechanism allows for faster initialization and higher training throughput compared to advanced LoRA schemes like PiSSA, LoRA-GA, or OLoRA.

Use MiSS when:

• You need faster initialization and higher training throughput than complex LoRA initialization schemes.
• You want a drop-in alternative to LoRA with minimal configuration changes.
• You want to balance expressive power with resource utilization.