EDS-NLP

EDS-NLP provides several core components designed for medical text processing. These components handle text cleaning, sentence boundary detection, and entity extraction using various matching strategies.

Key functionalities include:

• Text Normalization: Non-destructive input text normalization.
• Sentence Segmentation: Improved sentence boundary detection.
• Entity Extraction: Simple and powerful extraction using matchers, terminology lists, or conditional (contextual) logic.
• Text Classification: Unsupervised models to classify end lines.

EDS-NLP provides components that leverage Large Language Models (LLMs) deployed as external services to perform specialized information extraction tasks. These components are part of the edsnlp.pipes.llm module.

Available LLM-based components:

• eds.llm_markup_extractor: Extracts structured information from text by utilizing LLM-generated markup.
• eds.llm_span_qualifier: Predicts specific attributes or qualities of identified text spans using LLMs.

EDS-NLP provides trainable components designed to integrate machine learning models into biomedical information extraction pipelines. These components complement the standard rule-based pipeline components by allowing for model fitting and inference for tasks like NER, question answering, and entity linking.

All trainable components implement the TorchComponent class, ensuring a consistent API for both training and inference across the library.

A TorchComponent is a specialized component in EDS-NLP designed for deep learning operations on a spacy.tokens.Doc object. Unlike standard components, Torch components must be trained. They are used for tasks such as named entity recognition (NER), document classification, or predicting attributes of text spans.

To ensure efficiency during both training and inference, the TorchComponent architecture decouples deep learning logic from data handling by splitting the component's lifecycle into specific methods: preprocess, collate, forward, and postprocess.

from edsnlp.core.torch_component import TorchComponent

class MyComponent(TorchComponent):
    # Implementation follows the TorchComponent lifecycle
    ...

EDS-NLP is built on top of PyTorch. It acts as a companion library that provides specialized tools for preprocessing, composition, and evaluation of NLP models.

Trainable components in EDS-NLP, known as TorchComponents, are essentially PyTorch modules augmented with additional methods to handle feature preprocessing and postprocessing. Because of this architecture, EDS-NLP is fully compatible with the standard PyTorch ecosystem.

When processing large datasets, EDS-NLP uses lazy inference via the Stream object. Instead of executing operations immediately, operations (reading, converting, running pipes, writing) are scheduled in a Stream.

Execution only occurs when you call a terminal method like .execute(), .to_pandas(), or a writing method like .write_parquet().

A Stream consists of:

• A reader: The data source (e.g., file, database, list).
• stream.ops: A list of scheduled operations (functions, pipes, arguments).
• An optional writer: The data destination.
• An execution config: Defines the backend and resources (workers, batch size).

All stream methods (map(), map_batches(), map_gpu(), map_pipeline(), set_processing()) are chainable and return a new stream object.

A critical distinction in EDS-NLP is that trainable components are trained independently.

During training, a component does not automatically reuse the outputs of upstream components in the nlp pipeline. Instead, it reads the annotations it needs directly from the Doc objects in the training stream.

For example:

• If your classifier uses context_getter="sents[0:0] & words[-5:5]", you must explicitly add sentence boundaries to your training documents (e.g., via eds.sentences) so the classifier can access them.
• If your classifier relies on spans produced by an upstream component (like eds.dates), you must provide those spans explicitly in the training data (e.g., under a gold_spans group) because the upstream component's output is not automatically fed into the trainable component during the training loop.

EDS-NLP provides connectors to facilitate the conversion between different data formats and the internal spaCy representation used by the library. This allows you to import annotated data from external tools or export EDS-NLP results into formats compatible with other systems.

Currently supported connectors include:

• BRAT: For interacting with the BRAT annotation tool format.
• OMOP: For interacting with the OMOP Common Data Model format.

EDS-NLP is designed around a pipeline pattern where components typically qualify pre-extracted entities. A standard workflow involves three stages:

1. Normalization: Implement a normalizer (using eds.normalizer).
2. Entity Recognition: Add a component to find entities, such as the eds.matcher.
3. Entity Qualification: Add zero or more components to refine or qualify the found entities (e.g., eds.negation, eds.family, or eds.hypothesis). These qualifiers are used to detect false-positives.

This architecture allows you to build a pipeline that moves from raw text to highly qualified medical/clinical entities.

Depending on your evaluation requirements, you can choose between different granularity levels for NER metrics.