dedupe Python Library

dedupe is an open-source Python library that uses machine learning to perform de-duplication and entity resolution on structured data.

Key capabilities include:

• Machine Learning: Automatically creates optimum weights and blocking rules by reading human-labeled data.
• Efficiency: Designed to run on standard hardware (like a laptop) by making intelligent comparisons.
• Extensibility: Supports custom data types, string comparators, and blocking rules.
• Integration: Built as a library for easy integration into existing applications or import scripts.

Beyond the core library, you can use related tools:

• Dedupe.io: A full-service web service for de-duplicating and finding matches in messy data, providing a UI for cluster review and automation.
• csvdedupe: A command-line tool specifically designed for de-duplicating and linking CSV files.

Dedupe distinguishes between general deduplication and Record Linkage.

In the general case, an arbitrary number of records in a single dataset can refer to the same entity.

In Record Linkage, you are comparing two separate datasets to find matches between them. This scenario relies on the assumption that each dataset is individually unique. If this assumption holds, it implies:

1. Two records can only match if they originate from different datasets.
2. No other record in either dataset can match those two records.

To use the dedupe library in your Python projects for fuzzy matching, deduplication, and entity resolution, install it via pip:

pip install dedupe

Dedupe is a library rather than a standalone CLI tool. To learn how to implement it, you can use various example recipes tailored for different dataset sizes and use cases.

Detailed source code for these examples is available in the dedicated dedupe-examples repository. For interactive learning, you can view annotated walkthroughs online for specific scenarios like small data deduplication, record linkage, gazetteer usage, and database integrations (MySQL/Postgres).

For field types String, ShortString, Address, and Name, you can enable a Conditional Random Field (CRF) distance measure by setting crf=True. This measure provides more accurate results than the default edit distance but is significantly slower to compute.

Use this option when accuracy is more critical than processing speed for these specific field types.

dedupe.variables.String("name", crf=True)

Dedupe uses hierarchical clustering with centroid linkage to transform pairwise duplicate probabilities into groups of records. Because pairwise probabilities are not transitive (e.g., A matches B, and B matches C, but A might not match C), the library uses a centroid-based approach where all records within a certain distance of a group's centroid are assigned to that group.

To control the size and granularity of these groups, you must set a threshold for group membership, which defines the maximum distance from the centroid for a record to be included in a group.

You can install the dedupe library using pip to integrate machine learning-based de-duplication and entity resolution into your Python applications.

pip install dedupe

A variable definition is a collection of Variable objects that describes the fields you want to match. You provide these to the deduplication process to specify which record keys should be compared and how.

import dedupe.variables

[ 
    dedupe.variables.String("Site Name"),
    dedupe.variables.String("Address"),
    dedupe.variables.ShortString("Zip", has_missing=True),
    dedupe.variables.String("Phone", has_missing=True)
]