nervaluate

Unlike simple token-level evaluation, nervaluate uses entity-level evaluation to account for multi-token entities. It categorizes errors into five types:

• Correct (COR): Both boundaries and type match.
• Incorrect (INC): Output and gold annotation do not match.
• Partial (PAR): Boundaries are somewhat similar but not identical.
• Missing (MIS): A gold annotation is not captured by the system.
• Spurious (SPU): The system produces an entity that doesn't exist in the gold annotation.

These are measured using four evaluation schemas:

• Strict: Requires exact boundary match AND exact entity type.
• Exact: Requires exact boundary match, regardless of type.
• Partial: Allows partial boundary match, regardless of type.
• Type: Requires some overlap between the system tagged entity and the gold annotation.

Precision, recall, and F1-score are calculated differently depending on whether you are using an Exact Match schema (Strict, Exact) or a Partial Match schema (Partial, Type).

You can add support for new data formats by implementing a custom loader class. This is useful if your NER data is not in one of the supported formats (Nested lists, CoNLL style tab-delimited strings, or prodi.gy style lists of spans).

To implement a new loader:

1. Create a new class in nervaluate/loaders.py.
2. Inherit from the DataLoader base class.
3. Implement the load method. The load method must return a list of entity lists, where each entity is a dictionary containing the keys: label, start, and end.
4. Register the new loader by adding it to the _setup_loaders method in the Evaluator class.

Once registered, you can use the new format by passing the loader name to the loader argument when instantiating the Evaluator class.

# Conceptual implementation of a new loader
from nervaluate.loaders import DataLoader

class MyCustomLoader(DataLoader):
    def load(self, data):
        # Must return a list of entity lists
        # Each entity is: {'label': str, 'start': int, 'end': int}
        return [[{'label': 'ORG', 'start': 0, 'end': 5}]]

Nervaluate provides several strategies to score Named Entity Recognition (NER) predictions. Choosing a strategy depends on whether you want to penalize label mismatches or boundary inaccuracies.

1. StrictEvaluation

Requires an exact match of both boundaries (start and end) and the entity label.

• Correct: Perfect boundary and label match.
• Spurious: Predicted entity matches nothing.
• Missed: True entity matches nothing.
• Incorrect: Predicted entity overlaps a true entity but does not match perfectly.

2. PartialEvaluation

Focuses on boundary overlap and ignores labels.

• Correct: Perfect boundary match.
• Partial: Predicted entity has sufficient overlap with a true entity but boundaries differ.
• Spurious: Predicted entity matches nothing.
• Missed: True entity matches nothing.

3. EntityTypeEvaluation

Checks for label matches and uses overlap to find the best candidate.

• Correct: Sufficient overlap and matching label. If multiple true entities overlap, the one with the closest boundaries (minimum sum of start/end offsets) is chosen.
• Incorrect: Sufficient overlap but the label is wrong.
• Spurious: Predicted entity matches nothing.
• Missed: True entity matches nothing.

4. ExactEvaluation

Focuses on exact boundaries and ignores labels.

• Correct: Perfect boundary match, regardless of label.
• Incorrect: Predicted entity overlaps a true entity but boundaries differ.
• Spurious: Predicted entity matches nothing.
• Missed: True entity matches nothing.

The Evaluator class is the primary entry point for performing Named Entity Recognition (NER) model evaluation. It compares ground truth entities against predicted entities using various evaluation strategies.

Initialization Parameters

• true: The ground truth entities. Supports multiple formats (see Data Loaders).
• pred: The predicted entities. Supports multiple formats.
• tags: A List[str] of valid entity tags to be used during evaluation.
• loader: (Optional) The name of the loader to use. Options: "conll", "list", "dict". If set to "default", the loader is inferred from the input type.
• min_overlap_percentage: (Optional) A float (1-100) representing the minimum overlap required for a match to be considered a partial match.

from nervaluate import Evaluator

true_data = [...] # Your ground truth
pred_data = [...] # Your predictions
tags = ["PER", "ORG", "LOC"]

evaluator = Evaluator(
    true=true_data,
    pred=pred_data,
    tags=tags,
    loader="default",
    min_overlap_percentage=80.0
)

Use DictLoader to parse NER data provided as a list of documents, where each document is a list of dictionaries representing individual entities. This is the most direct way to load pre-structured entity data.

Input Format: List[List[Dict[str, Any]]] where each dictionary must contain the following keys:

• label: (str) The entity type.
• start: (int) The starting index.
• end: (int) The ending index.

Error Conditions:

• Raises ValueError if input is not a list.
• Raises ValueError if a document is not a list of dictionaries.
• Raises ValueError if an entity dictionary is missing label, start, or end.
• Raises ValueError if label is not a string or if start/end are not integers.

from nervaluate.loaders import DictLoader

loader = DictLoader()
dict_data = [[{"label": "PER", "start": 0, "end": 2}, {"label": "LOC", "start": 5, "end": 8}]]
entities = loader.load(dict_data)