LLM2Vec Documentation

LLM2Vec converts decoder-only LLMs into text encoders using a three-step process:

1. Enabling bidirectional attention: Modifying the model to allow bidirectional connections.
2. MNTP training: Training with Masked Next Token Prediction.
3. Unsupervised contrastive learning: Using objectives like SimCSE to refine embeddings.

This process allows decoder-only models to function as powerful text encoders for tasks like retrieval, classification, and clustering.

To host your project page on GitHub Pages:

1. Navigate to your repository Settings.
2. Select the Pages tab.
3. Under Source, set the branch to main and the folder to /docs.
4. Click Save.
5. Monitor the Actions tab to ensure the deployment job completes successfully.
6. Your site will be available at mcgill-nlp.github.io/<your-project-name>.

To train the model using Masked Next Token Prediction (MNTP), use the experiments/run_mntp.py script. This process is adapted from HuggingFace's Masked Language Modeling (MLM) approach. You must provide a JSON configuration file that specifies hyperparameters like model_name_or_path, dataset_name, mlm_probability, and LoRA settings.

python experiments/run_mntp.py train_configs/mntp/MetaLlama3.json

You can set up a project page using this template in two ways:

1. Using the GitHub Template

If you are starting a new repository, click the "Use this template" button on the GitHub repository page or use the template generator.

2. Cloning into an existing repository

If you already have a project repository, clone the template and copy the docs folder and README.md into your project root:

cd ..  # Go to parent directory
git clone https://github.com/McGill-NLP/project-page-template
cp -r project-page-template/docs my-project/
cp project-page-template/README.md my-project/docs/
cd my-project/

To perform SimCSE training, download the required Wikipedia dataset and place it in a cache directory. The script experiments/run_simcse.py uses the dataset_file_path defined in your configuration file to locate the data.

1. Download the dataset:

wget https://huggingface.co/datasets/princeton-nlp/datasets-for-simcse/resolve/main/wiki1m_for_simcse.txt

2. Organize the directory:

cache
└── wiki1m_for_simcse.txt

3. Run the training script with a configuration file:

python experiments/run_simcse.py train_configs/simcse/MetaLlama3.json

To use LLM2Vec, install the package from PyPI and then install flash-attn with --no-build-isolation. Alternatively, you can install the latest version by cloning the repository and using pip install -e ..

pip install llm2vec
pip install flash-attn --no-build-isolation

Supervised contrastive training uses the experiments/run_supervised.py script. This requires a dataset (such as the Echo embeddings dataset) placed in a cache directory. You can scale the training across multiple GPUs using torchrun by adjusting the --nproc_per_node argument.

Example directory structure:

cache
|── wiki1m_for_simcse.txt
└── echo-data
    ├── allnli_split1.jsonl
    ├── ...

Run training command:

torchrun --nproc_per_node=8 experiments/run_supervised.py train_configs/supervised/MetaLlama3.json

To tune the model for word-level tasks (like pos_tags, ner_tags, or chunk_tags), a classifier is trained on top of the model weights. The training process only updates the classifier weights. Use experiments/run_word_task.py for training and experiments/test_word_task.py for testing.

Example for Llama-2-7B on pos_tags:

python experiments/run_word_task.py train_configs/word-task/Llama2-bi-mntp.json
python experiments/test_word_task.py --config_file test_configs/word-task/Llama2-bi-mntp.json

Pages are stored as Markdown files in docs/_pages/.

To add a page:

1. Create a new file in docs/_pages/.
2. Set the permalink in the file's YAML front matter to your desired relative URL (e.g., permalink: /contact/ creates the URL mcgill-nlp.github.io/my-project/contact).
3. Add a corresponding entry in docs/_data/navigation.yml with the title and url.

To remove a page:

1. Delete the file in docs/_pages/.
2. Remove the matching entry from docs/_data/navigation.yml.

To evaluate models on the Massive Text Embedding Benchmark (MTEB), use the experiments/mteb_eval.py script.

Prerequisites: Install the evaluation dependencies:

pip install llm2vec[evaluation]

Usage: You must provide the model name, the specific task name, and a mapping file (task_to_instructions.json) that provides instructions for each task.

Example evaluation for the STS16 task:

python experiments/mteb_eval.py --model_name McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp-supervised \--task_name STS16 \--task_to_instructions_fp test_configs/mteb/task_to_instructions.json \--output_dir results

The docs/_config.yml file is used for global site configurations:

• Remove Dark Mode: Delete the dark_theme_css key from the file.
• Update Footer: Modify the footer settings directly within this file.

The navigation bar is controlled by the docs/_data/navigation.yml file. To add, remove, or modify links, edit this file using the following structure:

• title: The text displayed in the navbar.
• url: The relative URL of the page.

Note: It is recommended to avoid external URLs in the navbar; use the /home page for external links instead.