OpenNMT-py Documentation

OpenNMT-py is a PyTorch-based framework for Neural Machine Translation and Large Language Models (LLMs).

Key Capabilities:

• LLM Support: Converters for Llama (+ Mistral), OpenLlama, Redpajama, MPT-7B, and Falcon.
• Quantization: Support for 8-bit and 4-bit quantization, including LoRA adapters.
• Efficiency: Support for Multiquery attention (instead of Multihead) and tensor parallelism for models that exceed single GPU memory.
• Inference: Supports 4/8-bit inference and can be used with CTranslate2 for faster execution.

The MMLU (Massive Multitask Language Understanding) dataset used in this context consists of three primary data components:

• dev dataset: Used for few-shot learning to prime the model.
• test set: The source of evaluation questions.
• auxiliary_training data: Can be used for fine-tuning models that lack few-shot capabilities. This data is aggregated from other NLP multiple-choice datasets including MCTest, RACE, ARC, and OBQA.

Note on Knowledge Cutoff: Unless otherwise specified, questions refer to human knowledge as of January 1st, 2020. When prompting models, it may be necessary to specify that the questions are written for a 2020 audience.

To install OpenNMT-py from the source repository, clone the repository and run the setup script.

git clone https://github.com/OpenNMT/OpenNMT-py.git
cd OpenNMT-py
python setup.py install

Some advanced features, such as working with specific transforms or pretrained models, require additional packages. You can install these using the optional requirements file.

pip install -r requirements.opt.txt

Evaluations using the MMLU dataset in this project are performed on OpenNMT-py converted models.

Key differences from the original MMLU Hendrycks script:

• Instead of comparing logprobs of options (A, B, C, D), this implementation decodes the next token after the prompt.
• Token Handling:
  • Sentencepiece models: The next token can be 'A', 'B', 'C', 'D' or any other token.
  • BPE models: Tokens are often encoded with a leading space (e.g., ' A', ' B', ' C', ' D'). The evaluation script strips this leading space before computing metrics.

Note on Model Sizes:

• For 13B, 33B, and 40B models, evaluations are performed using the 4-bit loading option.

The simplest way to install OpenNMT-py is using pip.

pip install OpenNMT-py

To use multiple GPUs, first set the visible devices via environment variables:

export CUDA_VISIBLE_DEVICES=0,1,2,3

Then, specify -world_size and -gpu_ranks in your command. For example, to use 4 GPUs on a single node: python onmt_train.py -config config.yaml -world_size 4 -gpu_ranks 0 1 2 3

Multi-node training (Deprecated): To use multiple nodes, set -master_ip and -master_port. You must distribute the gpu_ranks across nodes and use -accum_count to minimize inter-node communication overhead.

Models converted using OpenNMT-py can be evaluated using the MMLU-FR dataset. The evaluation process utilizes the tkane script from the chain-of-thought-hub repository, which has been modified specifically to support OpenNMT-py models.

Results are typically reported as accuracy (ACC) across various subjects (e.g., ACC-abstract_algebra, ACC-anatomy, etc.) and an overall accuracy score (ACC-all).

Finetuning an LLM follows a similar workflow to training but requires specific configurations for the pretrained model (e.g., train_from, model_task: lm, encoder_type: transformer_lm).

Zero-out Prompt Loss

To ignore the prompt when calculating loss (useful for instruction tuning), add the insert_mask_before_placeholder transform and set zero_out_prompt_loss: true. You can customize the response_pattern used to locate the end of the prompt.

Run the finetuning process using python train.py with your configuration file.

You can use pretrained embeddings by configuring them in your YAML file. Supported types include GloVe and word2vec.

1. Ensure word_vec_size matches the dimensions of your pretrained file.
2. Use both_embeddings for both sides, or src_embeddings and tgt_embeddings for separate files.
3. Use freeze_word_vecs_enc or freeze_word_vecs_dec to prevent updating the embeddings during training.

# <your_config>.yaml

# embeddings will be used for both encoder and decoder sides
both_embeddings: glove_dir/glove.6B.100d.txt

# supported types: GloVe, word2vec
embeddings_type: "GloVe"

# word_vec_size need to match with the pretrained embeddings dimensions
word_vec_size: 100

You can control the sampling rate of different corpora by assigning a weight to each entry in the data configuration. The training process will sequentially take weight examples from each corpus.

Example: If corpus_1 has weight: 7 and corpus_2 has weight: 3, the trainer will sample 7 examples from the first and 3 from the second in each cycle.

# <your_config>.yaml

# Corpus opts:
data:
    corpus_1:
        path_src: toy-ende/src-train1.txt
        path_tgt: toy-ende/tgt-train1.txt
        weight: 7
    corpus_2:
        path_src: toy-ende/src-train1.txt
        path_tgt: toy-ende/tgt-train1.txt
        weight: 3
    valid:
        path_src: toy-ende/src-val.txt
        path_tgt: toy-ende/src-val.txt