BERTScore

Best Practices

• Reporting Results: Report the full hash code (e.g., roberta-large_L17_no-idf_version=0.3.0(hug_trans=2.3.0)-rescaled) in your research to ensure reproducibility, as changes in transformers versions can affect scores.
• Text Cleaning: Since RoBERTa uses a GPT2-style tokenizer, it is recommended to remove extra spaces using re.sub(r' +', ' ', sent) to avoid unexpected tokens.
• IDF: Using idf=True (or --idf in CLI) can improve correlation with human judgment, but be cautious if the reference set is very small.

Limitations

• Max Sequence Length: BERT, RoBERTa, and XLM use positional embeddings trained on sequences up to 512 tokens. BERTScore is undefined for sentences longer than 510 tokens (512 including [CLS] and [SEP]) and will truncate them. Consider using XLNet for longer inputs.

BERTScore values typically fall within a narrow range (e.g., 0.85 to 0.95 for RoBERTa-large) due to the nature of cosine similarities in contextual embeddings. To make scores more interpretable and map them to a more natural range (e.g., 0 to 1), you can use the rescaling feature available in version 0.3.0 and later.

Rescaling uses a linear transformation: $\hat{X} = \frac{X-Base}{1-Base}$, where Base is a pre-computed lower bound for a specific language and model. This operation preserves the correlation with human judgment (Pearson's $r$ and Kendall's $\tau$).

Requirements:

• You must specify the lang parameter to use this feature.
• The rescaling is applied after the BERTScore is computed.

out = bert_score.score(
    cands, refs, 
    rescale_with_baseline=True, lang="en"
)

To reproduce the system-level results for WMT17 as reported in the paper, download the necessary data using the provided shell script and then run the results script with a batch size of 16 and the roberta-large model.

bash download_wmt17.sh
python get_wmt17_sys_results.py -b 16 --model roberta-large

To prepare the data for tuning, run the download_data.sh script. This script downloads the WMT16 dataset and extracts it into a folder named wmt16. If the wmt16 folder already exists, the script will skip the download and extraction process.

bash download_data.sh

To reproduce the segment-level results for WMT18 as reported in the paper, download the necessary data using the provided shell script and then run the results script with a batch size of 16 and the roberta-large model.

bash download_wmt18.sh
python get_wmt18_seg_results.py -b 16 --model roberta-large

Use the get_rescale_baseline.py script to generate rescale baseline files for specific models. The resulting baseline files are saved in the rescale_baseline folder.

Arguments:

• --lang en: Specifies the language as English.
• -b 16: Sets the batch size (example uses 16).
• -m: Followed by the model names to be processed.

python get_rescale_baseline.py --lang en -b 16 -m \
    microsoft/deberta-large \
    microsoft/deberta-large-mnli

To compute rescale baselines for English models, you must first download the WMT17 English text dataset using the provided shell script.

bash download_text_data.sh

You can tune the best layer for multiple pre-trained models using the tune_layers.py script. Pass the model names as arguments to the -m flag.

Results (model name, best layer index, and Pearson correlation) are appended to best_layers_log.txt.

python tune_layers.py -m bert-base-uncased roberta-base albert-base-v2

After running the tuning script, you can use the results to update the default layer mappings in the library.

1. Open best_layers_log.txt and locate the last three lines (or the lines corresponding to your models).
2. The format is: 'model-name': layer_index, # correlation.
3. Copy these mappings and paste them into the model2layers dictionary in bert_score/utils.py to use these optimized layers in your BERTScore calculations.

'bert-base-uncased': 9, # 0.692518813886652
'roberta-base': 10, # 0.7062886932674598
'albert-base-v2': 9, # 0.6682362357086912

Install BERTScore via pip. Ensure you have Python >= 3.6 and PyTorch >= 1.0.0 installed.

To install the stable version from PyPI:

pip install bert-score

To install the latest unstable version from the master branch:

pip install git+https://github.com/Tiiiger/bert_score

To install from source:

git clone https://github.com/Tiiiger/bert_score
cd bert_score
pip install .

You can install bert_score using pip:

pip install bert_score

#!pip install bert_score

To enable rescaling in the command-line interface, use the --rescale_with_baseline flag. You must also provide the --lang flag to specify the language, as the baseline is language-dependent.

bert-score -r example/refs.txt -c example/hyps.txt \
           --lang en --rescale_with_baseline