Adapters Documentation

AdapterHub is a framework designed to simplify the integration, training, and usage of adapters and other efficient fine-tuning methods for Transformer-based language models.

It consists of two primary components:

1. Adapters: An add-on to Hugging Face's Transformers library that injects adapters into transformer models.
2. AdapterHub.ml: A central collection of pre-trained adapter modules.

Note: This documentation refers to the modern Adapters library. For the legacy adapter-transformers library, refer to https://docs-legacy.adapterhub.ml.

The PLBartAdapterModel is a BART-like sequence-to-sequence model designed for program and language understanding and generation. It is suitable for tasks such as:

• Code-summarization
• Code-generation
• Code-translation

The pre-trained plbart-base model is trained on multilingual denoising tasks involving Java, Python, and English, allowing it to learn program syntax, style (e.g., identifier naming conventions), and logical flow.

To use a pre-trained adapter, first initialize your Hugging Face model using adapters.init(model). Then, use the load_adapter() method to download and attach the adapter module.

By default, load_adapter() returns the name of the loaded adapter, which you can then pass to set_active_adapters() to enable it for model forward passes.

from transformers import BertModel
import adapters

# Initialize model for adapters
model = BertModel.from_pretrained('bert-base-uncased')
adapters.init(model)

# Load the adapter
adapter_name = model.load_adapter('sst-2')

# Activate the adapter
model.set_active_adapters(adapter_name)

AdapterFusion is trained in a second stage after training individual adapters on separate tasks. When setting up fusion, you must load the pre-trained adapters using model.load_adapter() before adding the fusion layer.

export GLUE_DIR=/path/to/glue
export TASK_NAME=SST-2

python run_fusion_glue.py \
  --model_name_or_path bert-base-uncased \
  --task_name $TASK_NAME \
  --do_train \
  --do_eval \
  --data_dir $GLUE_DIR/$TASK_NAME \
  --max_seq_length 128 \
  --per_device_train_batch_size 32 \
  --learning_rate 5e-5 \
  --num_train_epochs 10.0 \
  --output_dir /tmp/$TASK_NAME \
  --overwrite_output_dir

When working with CLIP in AdapterHub, use the CLIPModel class instead of CLIPAdapterModel. This is because CLIP does not come with pre-supported task-specific prediction heads.

Both the text and vision encoders within CLIP can be fitted with adapters. To specify which layers should receive adapters, use the leave_out parameter.

Important: Layer ID Calculation Layer IDs for CLIP are counted globally across both encoders, starting from the text encoder.

• Text Encoder IDs: Start from 0.
• Vision Encoder IDs: Follow immediately after the text encoder IDs.

Example: For a CLIP model with 12 layers in each Transformer encoder, the text encoder layers are IDs 0-11 and the vision encoder layers are IDs 12-23.

Vera is a LoRA-based method that uses frozen matrices $A$ and $B$ shared across all layers, reducing trainable parameters. It introduces trainable scaling vectors $b$ and $d$ that are multiplied by the frozen matrices.

Use VeraConfig to specify the initialization of scaling vectors and the frozen weights via the init_weights parameter.

from adapters import VeraConfig

config = VeraConfig()
model.add_adapter("vera_config", config=config)

After installing the core adapters package, navigate to the specific examples folder and install the required dependencies using the provided requirements.txt file.

pip install -r requirements.txt

To use the adapters library, clone the repository and install the package along with the specific requirements for your chosen example folder.

git clone https://github.com/adapter-hub/adapters
cd adapters
pip install .
pip install -r ./examples/pytorch/<your_examples_folder>/requirements.txt

Change your import namespace from transformers to adapters for all adapter-related classes. This affects:

• AdapterModel classes (e.g., AutoAdapterModel)
• Adapter configurations (e.g., PrefixTuningConfig)
• Adapter composition blocks (e.g., Stack)
• The AdapterTrainer class

The adapters repository uses a Makefile to manage testing and code quality. Ensure these pass before opening a pull request to avoid CI failures.

• Run all tests: Executes the full test suite.
• Auto format code: Runs black and isort to format code and imports.
• Run all quality checks: Runs black, isort, flake8, and additional custom consistency checks.

make test
make style
make quality