huggingface/evaluate

The Character Error Rate (CER) is computed using the following formula:

CER = (S + D + I) / N = (S + D + I) / (S + D + C)

Where:

• S: Number of substitutions
• D: Number of deletions
• I: Number of insertions
• C: Number of correct characters
• N: Total number of characters in the reference (N = S + D + C)

Note on values > 1.0: Because insertions (I) are included in the numerator but not the denominator (which is based on the reference length N), a high number of insertions can result in a CER greater than 1.0.

The EvaluationModule is the core base class for all evaluation logic in the library. It is extended by three specific types of modules:

1. Metric: Used for standard metric calculations.
2. Comparison: Used for comparative evaluations.
3. Measurement: Used for measurement-based evaluations.

Each of these functional modules has a corresponding information class (e.g., MetricInfo, ComparisonInfo, MeasurementInfo) that inherits from EvaluationModuleInfo to handle metadata and descriptive logic.

The FEVER metric consists of three main evaluation components that serve different purposes:

1. Label accuracy: Measures how often the predicted claim label (SUPPORTED, REFUTED, or NOT ENOUGH INFO) matches the gold label.
2. FEVER score: A strict metric that considers a prediction correct only if the label is correct and at least one complete gold evidence set is retrieved. Partial evidence retrieval results in a score of 0 for that claim.
3. Evidence F1: Computes the micro-averaged precision, recall, and F1 between predicted and gold evidence sentences, focusing on the quality of the retrieved evidence regardless of the label.

🤗 Evaluate categorizes its tools into three main types to cover different aspects of the machine learning pipeline:

• Metric: Evaluates model performance by comparing model predictions against ground truth labels (e.g., accuracy).
• Comparison: Used to compare two different models (e.g., comparing their agreement with ground truth).
• Measurement: Used to investigate and describe properties of a dataset itself.

All these types are accessed through the same unified entry point: evaluate.load.

The metric returns a dictionary containing the roc_auc score.

• For binary, multiclass, or multilabel (with average != None), the value is a single float: {'roc_auc': 0.778}

• For multilabel with average=None, the value is a numpy.array of floats (one per class): {'roc_auc': array([0.83333333, 0.375, 0.94444444])}

The Evaluator classes allow you to evaluate a triplet of model, dataset, and metric. The models are wrapped in a pipeline that handles preprocessing and post-processing.

Supported tasks include:

• text-classification (uses TextClassificationEvaluator)
• token-classification (uses TokenClassificationEvaluator)
• question-answering (uses QuestionAnsweringEvaluator)
• image-classification (uses ImageClassificationEvaluator)
• text-generation (uses TextGenerationEvaluator)
• text2text-generation (uses Text2TextGenerationEvaluator)
• summarization (uses SummarizationEvaluator)
• translation (uses TranslationEvaluator)
• automatic-speech-recognition (uses AutomaticSpeechRecognitionEvaluator)
• audio-classification (uses AudioClassificationEvaluator)

To run evaluations across multiple tasks simultaneously, use the EvaluationSuite.

Evaluating non-additive metrics (where $f(A \cup B) \neq f(A) + f(B)$, like F1 score) in a distributed environment is challenging because you cannot simply sum partial results.

🤗 Evaluate handles this by:

1. Performing distributed predictions/references on multiple nodes.
2. Storing these results temporarily in an Apache Arrow table to save memory.
3. Gathering all predictions and references to the first node when compute() is called.
4. Performing the final metric evaluation on that single node using the complete dataset.

This allows for high-speed distributed inference while maintaining the mathematical correctness of complex metrics.

R-squared is a statistical measure of how well a regression model fits the data.

Mathematical Logic: It is calculated as 1 - (SSR / SST), where:

• SSR (Sum of Squared Errors/Residuals): The sum of the squared differences between predicted and actual values.
• SST (Sum of Squared Total): The sum of the squared differences between actual values and the mean of the actual values.

Interpretation:

• 1.0: The model perfectly explains the variance.
• 0.0: The model explains none of the variance.
• Values between 0 and 1: Indicate the degree of variance explained (e.g., 0.75 means 75% of the variance is explained).

Limitations:

• It does not describe the nature of the relationship between variables.
• It is sensitive to irrelevant variables, which can lead to overfitting and artificially high R² values.
• It may be unreliable with small sample sizes.

The WER metric returns a float representing the average number of errors per reference word.

• A score of 0.0 indicates a perfect match between predictions and references.
• Lower values indicate better performance.
• A score of 1.0 (or higher in some cases depending on insertions) indicates no match or significant error.

Formula: WER = (S + D + I) / N Where:

• S: Substitutions
• D: Deletions
• I: Insertions
• N: Total words in reference (S + D + C)
• C: Correct words