DOCSEARCHbyAlgolia
Submit Docs

Crowd-Kit: Computational Quality Control for Crowdsourcing

repository·main·Indexed 42 days ago

https://github.com/toloka/crowd-kit

A Python library for computational quality control in crowdsourcing. It provides tools for aggregating categorical, pairwise, textual, and image segmentation responses using methods like Majority Vote, Dawid-Skene, and GLAD. The library includes metrics for uncertainty and agreement, built-in datasets for evaluation, and a learning subpackage with PyTorch-based layers like CrowdLayer and CoNAL for training models directly on noisy, non-aggregated crowd data.

Tokens
6.3K
Snippets
29
Records
36
Agent score
54%

What's inside Crowd-Kit

  1. Overview of Crowd-Kit capabilities

    main

    Crowd-Kit is a Python library designed for computational quality control in crowdsourcing. It provides tools for aggregating crowdsourced annotations and evaluating data quality.

    Key features include:

    • Aggregation Methods: Implementations for categorical, pairwise, textual, and segmentation responses.
    • Learning from Crowds: Deep learning methods and advanced aggregation algorithms (requires crowd-kit[learning]).
    • Metrics: Tools to measure uncertainty, consistency, and agreement with the aggregate.
    • Datasets: Loaders for popular crowdsourced datasets.

    The API is designed to be consistent with the scikit-learn pattern.

  2. Perform classification aggregation in Crowd-Kit

    main
    The crowdkit.aggregation.classification module provides tools for aggregating categorical responses from crowdsourced tasks. This is used when workers provide labels for a set of items, and you need to derive a consensus or estimate the true label for each item.

  3. Use multi-label aggregation methods

    main
    The crowdkit.aggregation.multilabel module provides aggregation methods specifically designed for multi-label response data. These methods are used to consolidate multiple labels provided by different workers into a single set of consensus labels for each item in a dataset.

  4. Use the crowdkit.learning module for learning from crowds

    main

    The crowdkit.learning module provides tools for training machine learning models using data collected from crowdsourcing tasks. This module is designed to handle various response types (categorical, multi-label, textual, etc.) and leverage crowd-sourced labels or gold standards to improve model performance.

    import crowdkit.learning

  6. Aggregate textual responses with crowd-kit

    main
    The crowdkit.aggregation.texts module provides tools for aggregating textual responses from crowdsourced tasks. This is used when workers provide free-text answers rather than categorical labels or numerical scores.

  7. Get started with categorical aggregation

    main

    To perform categorical aggregation (e.g., using the Dawid-Skene algorithm), follow these steps:

    1. Prepare your data: Load your annotations into a Pandas DataFrame. The DataFrame must contain exactly three columns: task, worker, and label.
    2. Initialize the aggregator: Use an aggregation class from crowdkit.aggregation.
    3. Predict labels: Use the .fit_predict() method (following the scikit-learn API) to obtain the aggregated labels.

    You can also use crowdkit.datasets.load_dataset() to download example datasets for testing.

    from crowdkit.aggregation import DawidSkene
from crowdkit.datasets import load_dataset
import pandas as pd

# Load your data (must have columns: task, worker, label)
df = pd.read_csv('results.csv') 

# Or use an example dataset:
# df, ground_truth = load_dataset('relevance-2')

# Aggregate using Dawid-Skene
aggregated_labels = DawidSkene(n_iter=100).fit_predict(df)

  8. Load example datasets with crowdkit.datasets

    main

    You can use the crowdkit.datasets module to load built-in example datasets for testing and development. The load_dataset function returns a tuple containing a pandas DataFrame (df) and the ground truth (ground_truth).

    Example usage:

    from crowdkit.datasets import load_dataset

df, ground_truth = load_dataset('relevance-2')

  9. Install Crowd-Kit

    main

    Install the core Crowd-Kit library using pip:

    pip install crowd-kit

    If you need to use the learning subpackage (which includes deep learning from crowds methods and advanced aggregation algorithms in PyTorch), install the optional learning dependencies:

    pip install crowd-kit[learning]

  10. Perform image segmentation aggregation

    main

    The crowdkit.aggregation.image_segmentation module provides tools for aggregating image segmentation tasks from crowdsourced workers. This is used when workers provide pixel-level annotations or masks for images.

    from crowdkit.aggregation import image_segmentation
# Use functions within this module to aggregate segmentation masks

  11. Train models using CrowdLayer and CoNAL

    main

    Instead of aggregating labels first, you can train machine learning models directly on raw, noisy crowd annotations using specialized layers. These layers are designed to be integrated into neural network architectures (like PyTorch Lightning modules).

    • CrowdLayer: Learns the confusion matrix $\mathbf{A}_w$ for every individual annotator $w$. It requires the number of intents (n_intents), the number of workers (n_workers), and a connection type (e.g., conn_type="mw").
    • CoNAL: Learns both annotator-specific confusion matrices $\mathbf{A}_w$ and one common confusion matrix $\mathbf{A}_g$.

    When using these layers during training, your data loader must provide the worker identifiers along with the task and label. During inference, the worker identifier is typically omitted.

    # Example initialization within a PyTorch module
from crowdkit.learning import CrowdLayer, CoNAL

# For CrowdLayer
self.CL = CrowdLayer(n_intents, n_workers, conn_type="mw")

# For CoNAL
self.CL = CoNAL(n_intents, n_workers)

  12. How Bradley-Terry aggregation works

    main

    Bradley-Terry models are used for pairwise ranking aggregation in subjective tasks (like search quality or personal preference) where a single 'correct' label might not exist.

    Instead of recovering a latent ground truth label, these models learn from the crowd by estimating weights for each item. In a pairwise comparison, the model learns the probability that one item is preferred over another.

    Crowd-Kit provides two main variations:

    • BradleyTerry: Implements marginalization maximization.
    • NoisyBradleyTerry: Uses L-BFGS-B optimization to account for noise, specifically estimating annotator biases and skills alongside item weights.