COMET

COMET is an open-source framework designed for Machine Translation (MT) evaluation. It serves two primary purposes:

1. Evaluating MT systems: Using existing high-performing metrics provided by the framework.
2. Metric development: Providing a foundation to train and develop new evaluation metrics.

For technical details on specific metrics, refer to the COMET Metrics section in the documentation.

COMET provides several types of models for MT evaluation:

• Reference-based Models: (e.g., Unbabel/wmt22-comet-da) Use source and reference to predict quality. Scores typically range from 0 to 1.
• Reference-free Models (QE): (e.g., Unbabel/wmt22-cometkiwi-da, Unbabel/wmt23-cometkiwi-da-xl, Unbabel/wmt23-cometkiwi-da-xxl) Predict quality using only the source and hypothesis. These are useful when no reference is available.
• eXplainable COMET (XCOMET): (e.g., Unbabel/XCOMET-XL, Unbabel/XCOMET-XXL) These models identify error spans and assign quality scores, allowing for error analysis based on the MQM typology (minor, major, or critical errors).

For older research compatibility (pre-2022), use Unbabel/wmt20-comet-da or Unbabel/wmt20-comet-qe-da.

COMET scores are typically normalized between 0 and 1:

• 1.0: Indicates a high-quality translation.
• 0.0: Indicates a translation that is no better than random chance.

While raw scores are useful for ranking translations and systems, you should use comet-compare when comparing two different systems to ensure the difference is statistically significant. For XCOMET models, scores can also be used to identify specific error types (minor, major, or critical) according to the MQM typology.

To run the toolkit's test suite using poetry and coverage, use the following commands. Note that testing on a CPU can be slow.

poetry run coverage run --source=comet -m unittest discover
poetry run coverage report -m # Expected coverage 76%

COMET supports UniTE (Unified Translation Evaluation) models. You can use the original checkpoints or Unbabel-trained variants:

• Unbabel/unite-mup: The original UniTE Metric.
• Unbabel/wmt22-unite-da: Trained on the same data as wmt22-comet-da, producing scores between 0 and 1.
• Unbabel/unite-xxl and Unbabel/unite-xl: Earlier curriculum phases of the XCOMET models.

Since 2021, the WMT Metrics task has used expert-based evaluation based on the Multidimensional Quality Metrics (MQM) framework. MQM annotations are available for the following years:

• 2020: 2020-mqm.tar.gz
• 2021: 2021-mqm.tar.gz
• 2022: 2022-mqm.tar.gz

Note: The original MQM data can be found at google/wmt-mqm-human-evaluation.

Install the unbabel-comet package using pip. Ensure you are using a Python 3 version of pip (e.g., pip3), as Python >= 3.5 is required.

pip install unbabel-comet

To train a new COMET metric, use the comet train CLI command and provide a configuration file in YAML format. COMET uses PyTorch Lightning for training, so all models are implemented as Lightning Modules.

Training Data Format

Your training data must be a CSV file containing the following columns:

• src: The source segment.
• mt: The machine translation hypothesis.
• ref: The reference segment.
• score: The human judgment score.

Example CSV structure:

src,mt,ref,score
Hello world!,Oi mundo.,Olá mundo!,0.5
This is a sample,este é um exemplo,isto é um exemplo!,0.8

comet train -f {my_configs}.yaml

To install COMET for standard use, ensure you have Python 3.8 or above and run the following commands:

pip install --upgrade pip
pip install unbabel-comet

Note: Some models (e.g., Unbabel/wmt22-cometkiwi-da) require you to acknowledge their license on the Hugging Face Hub and log in using the Hugging Face CLI.

You can train a custom COMET model using a configuration YAML file and then use it for scoring via the CLI. Once trained, you can also upload your model to the Hugging Face Hub to use it directly by its hub name.

# Train your own metric
comet-train --cfg configs/models/{your_model_config}.yaml

# Score using your trained checkpoint
comet-score -s src.de -t hyp1.en -r ref.en --model PATH/TO/CHECKPOINT