DOCSEARCHbyAlgolia
Submit Docs

recordlinkage toolkit

repository·master·Indexed 21 days ago

https://github.com/j535d165/recordlinkage

A modular Python toolkit for record linkage and deduplication in small to medium-sized datasets. It provides a high-performance framework leveraging pandas and numpy for indexing (blocking), comparing records using various similarity measures (string, date, numeric), and classifying matches via supervised (LogisticRegressionClassifier) or unsupervised (ECMClassifier, BernoulliEMClassifier) algorithms.

Tokens
28.2K
Snippets
96
Records
124
Agent score
75%

What's inside recordlinkage

  1. Getting started with RecordLinkage

    master

    RecordLinkage is a modular Python toolkit designed for record linkage and data deduplication tasks. To begin using the library, you should follow these primary workflows:

    1. Installation: Set up the environment.
    2. Basic Linking: Learn how to link two DataFrames using the provided guides.
    3. Data Deduplication: Apply the toolkit to identify duplicate records within a single dataset.

    The core workflow typically involves three main stages:

    • Preprocessing: Cleaning and standardizing data.
    • Indexing: Reducing the number of comparisons by creating candidate pairs.
    • Comparison: Calculating similarity scores between candidate pairs.
    • Classification: Using algorithms to decide if pairs are matches or non-matches.

  2. Understand classification algorithms in record linkage

    master

    In record linkage, classification is the process of dividing record pairs into matches and non-matches (distinct pairs). The Python Record Linkage Toolkit provides two main categories of algorithms:

    1. Supervised learning algorithms: These require training data (comparison vectors where the true match status is already known). They generally offer high accuracy and reliability. Supported algorithms include:

      • LogisticRegressionClassifier
      • NaiveBayesClassifier
      • SVMClassifier (Linear Support Vector Machines)

    2. Unsupervised learning algorithms: These do not require training data and are useful when match statuses are unknown. They are often more computationally intensive due to their iterative nature. Supported algorithms include:

      • KMeansClassifier
      • ECMClassifier (Expectation/Conditional Maximisation)

  3. How to implement a custom indexing algorithm

    master

    You can create custom indexing logic by subclassing recordlinkage.base.BaseIndexAlgorithm.

    Depending on your use case, you must overwrite one of the following methods:

    1. For linking two datasets: Overwrite _link_index(self, df_a, df_b). This method receives two pandas.Series (or tuples of Series) and must return a 2-level pandas.MultiIndex. The names of the MultiIndex must correspond to the index names of df_a and df_b respectively.
    2. For deduplication (single dataset): Overwrite _dedup_index(self, df_a). This method receives a single dataset and returns the index of pairs.

    Note: If you use the standard _link_index logic for deduplication, the base class automatically handles removing self-pairs and duplicate combinations (e.g., it ensures only one of (i, j) or (j, i) is returned).

    from recordlinkage.base import BaseIndexAlgorithm
import pandas as pd

class MyCustomIndex(BaseIndexAlgorithm):
    def _link_index(self, df_a, df_b):
        # Logic to find pairs
        # Must return a pandas.MultiIndex
        return pd.MultiIndex.from_product(
            [df_a.index.values, df_b.index.values],
            names=[df_a.index.name, df_b.index.name]
        )

  4. Apply phonetic encoding for better linkage performance

    master

    Phonetic algorithms index words based on their pronunciation (e.g., the Soundex algorithm). You should apply phonetic encoding before the indexing and comparing steps to improve performance and accuracy in most situations.

    Use the recordlinkage.preprocessing.phonetic function to apply these algorithms to your data.

    from recordlinkage.preprocessing import phonetic

# Example usage (conceptual)
# phonetic(series, algorithm='soundex')

  5. The Record Linkage Workflow

    master

    The Python Record Linkage Toolkit follows a standard workflow for linking records or performing deduplication. The process consists of five main steps:

    1. Cleaning: Standardizing and cleaning data.
    2. Indexing: Creating a set of candidate record pairs (e.g., using blocking or sorted neighbourhood indexing) to avoid the computational cost of comparing every single record against every other record.
    3. Comparing: Using comparison and similarity measures to generate comparison vectors for the candidate links.
    4. Classifying: Applying supervised (e.g., Logistic Regression) or unsupervised (e.g., Expectation-Maximization) algorithms to determine if pairs are matches.
    5. Evaluation: Assessing the quality of the linkage results.

    The toolkit is designed to work extensively with pandas and numpy for efficient data manipulation.

  6. Use experimental algorithms in recordlinkage.contrib

    master

    The recordlinkage.contrib module contains experimental algorithms and contributions that are not officially supported. These features are intended for testing and may undergo interface changes or be removed without notice. They are often candidates for future inclusion in the core toolkit.

    Commonly used experimental imports include:

    • Indexing algorithms: recordlinkage.contrib.index.NeighbourhoodBlock
    • Comparison algorithms: recordlinkage.contrib.compare.random.RandomContinuous
    from recordlinkage.contrib.index import NeighbourhoodBlock

# or 

from recordlinkage.contrib.compare.random import RandomContinuous

  7. Classify record pairs using built-in classifiers

    master

    Classification in recordlinkage is the process of categorizing record pairs into matches, non-matches, or possible matches. The toolkit provides several built-in algorithms categorized by whether they require training data:

    Supervised Classifiers

    These require labeled training data to learn the relationship between features and matches:

    • recordlinkage.LogisticRegressionClassifier
    • recordlinkage.NaiveBayesClassifier
    • recordlinkage.SVMClassifier

    Unsupervised Classifiers

    These do not require training data and can find patterns in the data automatically:

    • recordlinkage.ECMClassifier (Expectation-Maximization)
    • recordlinkage.KMeansClassifier

  8. Create custom comparison algorithms by subclassing BaseCompareFeature

    master

    To implement a custom comparison logic, subclass recordlinkage.base.BaseCompareFeature and overwrite the _compute_vectorized method. The _compute_vectorized method must accept the series from the two datasets and return a pandas.Series representing the similarity scores.

    Implementation steps:

    1. Inherit from BaseCompareFeature.
    2. Implement _compute_vectorized(self, s1, s2).
    3. Use the .add() method on a recordlinkage.Compare object to include your custom feature.

    Note: The recordlinkage.Compare class selects the columns specified by the labels before passing them to your custom algorithm.

    from recordlinkage.base import BaseCompareFeature

class CustomFeature(BaseCompareFeature):
    def _compute_vectorized(self, s1, s2):
        # algorithm that compares s1 and s2
        # return a pandas.Series
        return ... 

feat = CustomFeature()
# Use it with a comparer
comparer = rl.Compare()
comparer.add(feat)

  9. Run tests for the recordlinkage package

    master

    To ensure your modifications work correctly, you should run the existing test suite using pytest. First, install pytest via pip, then execute the tests from the package directory using the python -m pytest command targeting the tests/ directory.

    pip install pytest
python -m pytest tests/

  10. How to link records using the toolkit

    master

    To perform record linkage, you typically follow a pattern of indexing, comparing, and then classifying.

    1. Setup and Data Loading

    Import recordlinkage and pandas, then load your datasets into pandas DataFrames.

    2. Indexing (Candidate Generation)

    Use recordlinkage.Index() to create candidate links. A common technique is blocking, which only considers pairs that agree on a specific attribute (e.g., 'surname').

    3. Comparison

    Use recordlinkage.Compare() to define which attributes to compare and which similarity measures to use (e.g., string, exact). The compute() method generates comparison vectors.

    4. Classification

    You can use supervised learning (requiring training/golden data) or unsupervised learning (like the ECM algorithm) to classify the links.

    import recordlinkage
import pandas

# 1. Load data
df_a = pandas.DataFrame(YOUR_FIRST_DATASET)
df_b = pandas.DataFrame(YOUR_SECOND_DATASET)

# 2. Indexing (Blocking)
indexer = recordlinkage.Index()
indexer.block('surname')
candidate_links = indexer.index(df_a, df_b)

# 3. Comparison
compare = recordlinkage.Compare()
compare.string('name', 'name', method='jarowinkler', threshold=0.85)
compare.exact('sex', 'gender')
compare_vectors = compare.compute(candidate_links, df_a, df_b)

# 4. Classification (Supervised Example)
true_linkage = pandas.Series(YOUR_GOLDEN_DATA, index=pandas.MultiIndex(YOUR_MULTI_INDEX))
logrg = recordlinkage.LogisticRegressionClassifier()
logrg.fit(compare_vectors[true_linkage.index], true_linkage)
predictions = logrg.predict(compare_vectors)

# 4. Classification (Unsupervised Example)
ecm = recordlinkage.BernoulliEMClassifier()
ecm.fit_predict(compare_vectors)

  12. Handle missing values in comparison vectors

    master

    Most classification algorithms cannot handle comparison vectors containing missing values. To prevent errors, it is a common practice in record linkage to convert missing values into disagreeing comparisons.

    When loading datasets for classification, use the missing_values=0 argument in the data loading function (e.g., load_krebsregister) to ensure missing values are treated as non-matches.

    import recordlinkage as rl
from recordlinkage.datasets import load_krebsregister

# Convert missing values to 0 (disagreeing comparisons)
krebs_X, krebs_true_links = load_krebsregister(missing_values=0)