AlignScore: Automatic Factual Consistency Evaluation Metric

AlignScore is a metric designed for the automatic evaluation of factual consistency in text pairs. It evaluates whether the information in a candidate text (b) is contained within a source text (a) without contradiction.

It is particularly useful for tasks like:

• Summarization: Checking if a generated summary is faithful to the original document.
• Paraphrase: Verifying if a paraphrase maintains the original meaning.
• Dialog: Ensuring responses in a conversation are factually grounded.

AlignScore provides two main model variants:

• AlignScore-large: Based on RoBERTa-l (355M parameters).
• AlignScore-base: Based on RoBERTa-b (125M parameters).

AlignScore is benchmarked against both traditional metrics and LLM-based metrics across several datasets.

Comparison with similar-sized metrics

AlignScore (especially the large version) outperforms many popular metrics like ROUGE, BLEU, BERTScore, and SummaC on benchmarks such as SummaC and TRUE. Performance is reported using AUC ROC for SummaC/TRUE and Spearman Correlation for other datasets (e.g., XSumFaith, SummEval, SamSum).

Comparison with LLM-based metrics

AlignScore is compared against GPT-based metrics (like G-EVAL and GPTScore). While LLM-based metrics can be powerful, they are often noted to be more expensive and slower than AlignScore. AlignScore-large shows competitive Spearman Correlation coefficients on datasets like SummEval, QAGS-XSUM, and QAGS-CNNDM.

AlignScore requires PyTorch 1.12.1 for reproduction of results. Follow these steps to install:

1. Install PyTorch 1.12.1.
2. Clone the repository and install via pip:

   pip install .

3. Download the required spaCy model for sentence tokenization:

   python -m spacy download en_core_web_sm

The Inferencer supports specific modes for Natural Language Generation (NLG) evaluation by setting the nlg_eval_mode attribute. This determines which model head is used and how scores are aggregated.

Modes:

• 'reg': Uses the regression head.
• 'bin': Uses the binary relationship head.
• 'nli': Uses the triple/NLI head.
• 'reg_sp', 'bin_sp', 'nli_sp': Sentence-pair modes that use inference_example_batch for SummaC-style aggregation (calculating scores per sentence pair and aggregating).

To use these, set the attribute on your Inferencer instance before calling nlg_eval(premise, hypo).

inferencer.nlg_eval_mode = 'nli_sp'  # Example: NLI sentence-pair mode
reg, bin, tri = inferencer.nlg_eval(premise, hypo)

When initializing the AlignScore class, you can provide the following parameters:

• model: The backbone model. Currently, only RoBERTa is supported (e.g., 'roberta-base').
• batch_size: The batch size for inference.
• device: The device to run the metric on (e.g., 'cuda:0').
• ckpt_path: The file path to the model checkpoint.
• evaluation_mode: Determines the classification head and splitting method. Options include:
  • 'nli_sp': (Default) Uses 3-way NLI classification with chunk-sentence splitting.
  • 'nli': Uses 3-way NLI classification.
  • 'bin_sp': Uses binary classification with chunk-sentence splitting.
  • 'bin': Uses binary classification.

To evaluate whether a claim is factually consistent with a context, use the score method of the AlignScore class.

Inputs:

• contexts: A list of context strings.
• claims: A list of claim strings.

Output: A single numerical value representing the degree of factual consistency.

from alignscore import AlignScore

scorer = AlignScore(model='roberta-base', batch_size=32, device='cuda:0', ckpt_path='/path/to/checkpoint', evaluation_mode='nli_sp')
score = scorer.score(contexts=['hello world.'], claims=['hello world.'])

The BERTAlignModel uses PyTorch Lightning's save_hyperparameters(). When configuring the optimizer via configure_optimizers, the following hyperparameter keys are expected to be present in the model's hparams:

• learning_rate: Learning rate for the AdamW optimizer.
• adam_epsilon: Epsilon for AdamW.
• weight_decay: Weight decay value.
• warmup_steps_portion: A float representing the portion of total training steps used for linear warmup.

AlignScore provides two versions of pre-trained checkpoints:

• AlignScore-base: Based on RoBERTa-base (125M parameters). URL: https://huggingface.co/yzha/AlignScore/resolve/main/AlignScore-base.ckpt
• AlignScore-large: Based on RoBERTa-large (355M parameters). URL: https://huggingface.co/yzha/AlignScore/resolve/main/AlignScore-large.ckpt

If you want to train an alignment model from scratch or on custom data, use the train.py script.

Example command:

python train.py --seed 2022 --batch-size 32 \
--num-epoch 3 --devices 0 1 2 3 \
--model-name roberta-large -- ckpt-save-path ./ckpt/ \
--data-path ./data/training_sets/ \
--max-samples-per-dataset 500000

Arguments:

• --seed: Random seed for initialization.
• --batch-size: Batch size for training.
• --num-epoch: Number of training epochs.
• --devices: List of GPU IDs for training (e.g., 0 1 2 3).
• --model-name: Backbone model name (default: roberta-large).
• --ckpt-save-path: Path to save the checkpoint.
• --training-datasets: Names of the training datasets.
• --data-path: Path to the training datasets.
• --max-samples-per-dataset: Maximum number of samples to take from each dataset.

To benchmark AlignScore against various datasets (like TRUE, SummaC, etc.), use benchmark.py.

Prerequisites:

1. Install dependencies: pip install -r requirements.txt.
2. Some dependencies must be downloaded manually (check python benchmark.py --help).
3. Note: Installing summac may cause dependency conflicts. If this happens, reinstall alignscore to restore correct versions.

AlignScore Evaluation Arguments:

• --alignscore: Enable evaluation of the AlignScore metric.
• --alignscore-model: Backbone model name ('roberta-base' or 'roberta-large').
• --alignscore-ckpt: Path to the saved checkpoint.
• --alignscore-eval-mode: Evaluation mode (defaults to nli_sp).
• --device: Device to use (defaults to cuda:0).
• --tasks: Specific tasks to benchmark (e.g., SummEval, QAGS-CNNDM).

The Inferencer class is the primary entry point for running AlignScore evaluations. You can initialize it with a checkpoint path to load a trained model, or without one to load an untrained model (which will trigger a warning).

Parameters:

• ckpt_path: Path to the model checkpoint. If None, an untrained model is loaded.
• model: The base model name (default: 'bert-base-uncased').
• batch_size: Number of examples to process in one batch (default: 32).
• device: Device to run inference on (default: 'cuda').
• verbose: Whether to display progress bars (default: True).

from alignscore.inference import Inferencer

inferencer = Inferencer(ckpt_path='path/to/checkpoint.ckpt', batch_size=16, device='cuda')