sacreBLEU

To use a variable number of reference segments per hypothesis, you can replace a missing reference segment with either None or an empty string '' within the refs list.

When using this pattern, the metric signature returned by get_signature() will reflect this by setting nrefs:var instead of a fixed number.

from sacrebleu.metrics import BLEU

# The first sentence has no reference (using empty string)
refs = [
    ['', 'It was not unexpected.', 'The man bit him first.'],
    ['The dog had bit the man.', 'No one was surprised.', 'The man had bitten the dog.'],
]
sys = ['The dog bit the man.', "It wasn't surprising.", 'The man had just bitten him.']

bleu = BLEU()
bleu.corpus_score(sys, refs)
# Signature will show nrefs:var
print(bleu.get_signature())

When running significance tests, you must define a baseline to compare other systems against. SacreBLEU supports two main ways to handle this:

1. Using --input/-i (Recommended): The first system file path provided to the --input/-i flag is automatically designated as the baseline system. SacreBLEU will automatically discard the baseline file if it also appears in the list of other systems (e.g., using a wildcard like -i baseline.txt systems/*.txt).
2. Using Tab-Separated Input (Not Recommended): If you redirect a tab-separated input file to SacreBLEU, the first column of hypotheses is treated as the baseline. However, this is not recommended because systems will be enumerated numerically (1 to N) rather than using human-readable names.

Note: Significance tests compare the baseline against every other system provided.

Since version 2.0.0, the default output format is json. If your downstream tools expect the old textual output, you have two options:

1. Pass the flag explicitly: Use -f text in your command.
2. Set an environment variable: Export SACREBLEU_FORMAT=text in your shell configuration to make this change persistent.

To parse the JSON output (the new default) using jq, you can target the score key directly:

sacrebleu -i output.detok.txt -t wmt17 -l en-de | jq -r .score

Install the official Python module from PyPI. Note that Python >= 3.9 is required.

To install standard support:

pip install sacrebleu

To install additional tokenizer support for specific languages, use the following commands:

• Japanese support (via mecab-python3):

  pip install "sacrebleu[ja]"

- **Korean support** (via `pymecab-ko`):
  ```bash
pip install "sacrebleu[ko]"

To study the translationese effect, you can filter the evaluation to a specific subset of sentences based on their original language using the --origlang flag.

Note: You must pass the entire translated test set to the command; sacreBLEU will only return the score for the requested subset.

You can score translations by providing the hypothesis file (the output of your system) via the -i or --input flag, or by redirecting it to STDIN. If using a built-in test set, provide the test set name with -t and the language pair with -l.

Important: Ensure your input files are detokenized.

Since version 2.0.0, SacreBLEU can evaluate multiple system output files simultaneously for a specific test set and language pair. This produces a formatted table comparing all systems across the requested metrics.

Use the -t flag for the test set, -l for the language pair, and -i to pass one or more system files (supports shell globs).

$ sacrebleu -t wmt17 -l en-de -i newstest2017.online-* -m bleu chrf

SacreBLEU supports evaluating a system against multiple reference files.

Method 1: Positional Arguments (Recommended) Pass all reference files as positional arguments before the -i (system) flag.

Method 2: Tab-delimited file (Less recommended) Concatenate references using tabs as delimiters into a single file. If using this method, you must provide the --num-refs or -nr flag to specify the number of references per line.

# Method 1: Positional arguments
$ sacrebleu ref1 ref2 -i system -m bleu chrf ter

# Method 2: Tab-delimited file
$ paste ref1 ref2 > refs.tsv
$ sacrebleu refs.tsv --num-refs 2 -i system -m bleu

To estimate the statistical significance of a score, use the --confidence flag. This enables bootstrap resampling and outputs:

1. The actual system score.
2. The true mean ($\mu$) estimated from bootstrap resampling.
3. The 95% confidence interval around the mean.

Configuration:

• The default number of bootstrap resamples is 1000. Use --confidence-n to change this.
• Use --short with -f text for a condensed output format.

$ sacrebleu -t wmt17 -l en-de -i output.detok.txt -m bleu chrf --confidence -f text --short

sacreBLEU can automatically download and unpack common test sets when requested via the -t or --test-set flag. This is useful for scoring against standard benchmarks like WMT. You can also use --echo to extract specific parts of a test set, such as the source text or metadata.

To download the source text for a test set and save it to a file:

$ sacrebleu -t wmt17 -l en-de --echo src > wmt17.en-de.en

You can also score against pre-existing system outputs in specific test sets (e.g., wmt21/systems) by using --echo to specify the system name:

$ sacrebleu -t wmt21/systems -l zh-en --echo NiuTrans

# Example: Download source and score via pipe
$ sacrebleu -t wmt17 -l en-de --echo src > wmt17.en-de.en
$ cat wmt17.en-de.en | translate.sh | sacrebleu -t wmt17 -l en-de

The random seed used for bootstrap resampling and randomization tests can be controlled via the SACREBLEU_SEED environment variable.

• If SACREBLEU_SEED is set to a numeric string (e.g., '12345'), the RNG will be fixed to that value, ensuring reproducible results.
• If SACREBLEU_SEED is set to 'none' (case-insensitive), the seed will not be fixed, and entropy will be pulled from the OS.
• If the environment variable is not set, it defaults to '12345'.