repository·main·Indexed 21 days ago
https://github.com/norskregnesentral/skweakA framework for weak supervision in NLP designed to build high-quality NER models without manually labeled data. It aggregates multiple noisy supervision sources—such as spaCy models, gazetteers, and custom heuristic functions—using Hidden Markov Models (HMM) or majority voting to produce unified labels. The toolkit includes specialized annotators for token and span constraints, document-level consistency, and utilities for exporting aggregated annotations to spaCy DocBin format.
You can install the core skweak library using pip. If you want to install directly from the GitHub repository, use the git URL. Note that some examples and tests may require trained spaCy pipelines, which can be downloaded separately.
# Install core library
pip install skweak
# Install from repo
pip install --user git+https://github.com/NorskRegnesentral/skweak
# Download a spaCy model if needed
python -m spacy download en_core_web_smAfter applying multiple labelling functions, use an aggregation model to resolve conflicts and produce a single set of annotations.
skweak.aggregation.HMM (Hidden Markov Model) can be used to estimate a generative model. You can define 'underspecified labels'—labels that represent a group of more specific labels (e.g., an ORG label that could be either a COMPANY or an OTHER_ORG).
Use .fit_and_aggregate(docs) to train the model on your annotated corpus and update the Doc objects with the aggregated results.
import skweak
# Define the model with the specific labels you want to predict
model = skweak.aggregation.HMM("hmm", ["COMPANY", "OTHER_ORG"])
# Map an underspecified label to a set of specific labels
model.add_underspecified_label("ORG", ["COMPANY", "OTHER_ORG"])
# Fit the model and update the docs
docs = model.fit_and_aggregate(docs)The AbstractAnnotator is the base class for all annotation or aggregation sources in skweak.
Core Interface:
__init__(name: str): Every annotator must have a unique name used to store its results in doc.spans[name].__call__(doc: Doc) -> Doc: The primary method to annotate a single spaCy Doc.pipe(docs: Iterable[Doc]) -> Iterable[Doc]: The preferred method for processing large batches of documents. The default implementation iterates through the stream and calls __call__ on each document.An AbstractAggregator is an AbstractAnnotator that combines multiple weak supervision sources (labeling functions) into a single prediction.
Lifecycle and Workflow:
name and a list of labels (the target output labels)..fit(docs) to learn parameters from a collection of documents. Use .fit_and_aggregate(docs) to perform both steps in one go.Doc (via __call__), it:doc.spans (filtering out those marked aggregated=True or avoid_in_aggregation=True).get_observation_df.aggregate logic.doc.spans[self.name].probs (the full probability distribution), aggregated (set to True), and sources (the list of contributing sources).Label Groups: You can use add_label_group(coarse_label, sub_labels) to handle underspecified labels. For example, if a labeling function returns ENT, you can map it to [PER, ORG] so the aggregator treats a vote for ENT as a partial vote for its constituents.
The GenerativeModelMixin is a base class for implementing aggregation methods based on generative models (where states are latent 'true' labels and observations are predictions from labeling sources).
Note: This class should not be instantiated directly; it is intended to be subclassed (e.g., by NaiveBayes or HMM).
Key features include:
initial_weights (a dictionary mapping source names to weights in range [0, +inf)) and a redundancy_factor to penalize correlated labeling functions.aggregate(obs) method takes a 2D DataFrame of observations and returns a 2D DataFrame of label probabilities._fit(all_obs, ...) method implements the Expectation-Maximization (EM) algorithm to learn model parameters from a collection of observations.class MyCustomAggregator(GenerativeModelMixin, SomeOtherMixin):
def get_posteriors(self, X: Dict[str, np.ndarray]) -> np.ndarray:
# Implement posterior inference logic
pass
def _reset_counts(self, sources):
# Implement parameter reset logic
pass
def _add_mv_counts(self, all_obs):
# Implement initial count logic from majority voter
pass
def _accumulate_statistics(self, X: Dict[str, np.ndarray]) -> float:
# Implement E-step statistics accumulation
pass
def _do_mstep_latent(self):
# Implement M-step for latent parameters
passWhen performing token-level sequence labeling (e.g., NER with BIO/BILOU schemes), use a class that inherits from SequenceAggregatorMixin.
Key Features:
IO, BIO, BILUO, and BILOU prefixes. You specify this via the prefixes argument during initialization.[ORG] and prefixes are BIO, it manages B-ORG, I-ORG, and O).utils.spans_to_array to convert the document's spans into a token-level array for the aggregator.When performing span-level classification (where you predict labels for specific text segments), use a class that inherits from TextAggregatorMixin. This mixin provides a standard implementation for get_observation_df, _get_spans, and _get_probs based on unique spans found in the document.
It expects the input Doc to have spans from various sources that share identical boundaries.
For tasks where multiple labels can be true for the same span or token, use MultilabelAggregatorMixin.
How it works:
It decomposes the multi-label problem into multiple single-label sub-problems. For each target label, it creates a dedicated sub-model that treats the task as a binary choice: [label] vs [NOT/label] (or [prefix-label] vs [prefix-O] for sequence labeling).
Key Methods:
set_exclusive_labels(exclusive_labels: Set[str]): Defines a set of labels that cannot co-occur. The aggregator will ensure that probabilities for these labels are normalized so they do not exceed 1.0 when combined with the 'null' label.Once you have aggregated your labels, you can convert the skweak annotations back into standard spaCy DocBin format to train a final production model.
hmm key) to the standard doc.ents attribute.skweak.utils.docbin_writer(docs, path) to save the data.# Transfer aggregated spans to standard spaCy entities
for doc in docs:
doc.ents = doc.spans["hmm"]
# Save to a spaCy DocBin file
skweak.utils.docbin_writer(docs, "../data/reuters_small.spacy")A typical workflow for using skweak for sentiment analysis or NER involves these steps:
DocBin objects. Store ground truth labels in doc.user_data["gold"] if available.DocBOWAnnotator, BERT-based models) to generate predictions. These predictions are saved into new DocBins.skweak.aggregation (like HMM or MajorityVoter) to combine the predictions from all weak labelers into a single set of labels.f1_score.All majority voting aggregators accept an initial_weights argument.
Optional[Dict[str, float]][0, +inf).None, all sources are assumed to have a weight of 1.0 in the dictionary.The skweak workflow follows a four-step process to generate labels from unlabelled text:
Doc objects.Doc objects and annotate spans with labels. LFs can be heuristics, gazetteers, or machine learning models.