DOCSEARCHbyAlgolia
Submit Docs

Splink Documentation

repository·master·Indexed 25 days ago

https://github.com/moj-analytical-services/splink

A high-performance Python library for probabilistic record linkage and deduplication based on the Fellegi-Sunter model. Splink enables the identification and clustering of records referring to the same entity across datasets without unique identifiers. It supports multiple backends including DuckDB, Spark, and PostgreSQL, and provides tools for training, inference, and clustering via the Linker class.

Tokens
45.8K
Snippets
90
Records
247
Agent score
78%

What's inside Splink

  1. What is Splink and when should you use it?

    master

    Splink is a Python package for probabilistic record linkage (entity resolution). It is used to deduplicate and link records from datasets that lack unique identifiers.

    Best Use Cases

    Splink works best with input data containing multiple columns that are not highly correlated.

    • Good examples: A person dataset with full name, date of birth, and city. A company dataset with name, turnover, and sector.
    • Bad examples: A single column containing a 'bag of words' (e.g., just a company name column with no other details). Data where columns are highly predictable from one another (e.g., city being highly predictable from postcode).

    Key Capabilities

    • Speed: Can link ~1 million records on a laptop in about a minute.
    • Scalability: Supports DuckDB (Python) and big-data backends like Spark (for 100+ million records).
    • Unsupervised: No training data is required.
    • Accuracy: Supports term frequency adjustments and custom fuzzy matching logic.

  2. Explore the Splink User Guide topics

    master

    The Splink User Guide provides in-depth documentation on record linkage theory, model construction, and performance optimization. It is organized into several key functional areas:

    • Record Linkage Theory: Theoretical foundations and intuition for model parameters.
    • Linkage Models: Building blocks of a Splink model, supported SQL backends, and using Splink Settings dictionaries.
    • Data Preparation: Feature engineering techniques to improve model accuracy.
    • Blocking: Using Blocking Rules to reduce the number of comparisons.
    • Comparing Records: Defining Comparison objects, using string comparators for fuzzy matching, and applying Term Frequency Adjustments for skewed data.
    • Evaluation: Methods for evaluating models, links, and clusters (including Clerical Labelling).
    • Performance: Strategies for making Splink models run more efficiently.

  3. Explore the Splink API structure

    master
    The Splink API is organized into several functional areas. The primary entry point for most workflows is the Linker class, which manages the linkage process through training, inference, clustering, and evaluation. Other key components include the Comparison and Comparison Level libraries for defining how columns are compared, and SplinkDataFrame for handling data within the Splink ecosystem.

  4. Key improvements in Splink 4

    master

    Splink 4 is an incremental improvement over version 3 designed to make the library easier to use without changing core workflows (training and predicting).

    Ease of Use

    • Reduced Boilerplate: Users write less code to achieve the same results.
    • Intuitive Imports: Function imports are simpler and better grouped.
    • Pythonic Configuration: Settings and configuration can be constructed using Python objects, enabling better IDE autocomplete and reducing reliance on remembering string-based setting names.
    • Reduced Dialect Specificity: Less code required for specific database dialects.

    Compatibility

    • Workflow Stability: The core steps to train a model and predict results remain the same.
    • Model Compatibility: Models trained in Splink 3 are compatible with Splink 4.

  5. Design a multi-stage linkage pipeline with Splink

    master

    When using Splink in a production environment, it is recommended to follow a modular pipeline pattern rather than running a single monolithic process. This improves idempotency, allows for easier debugging, and decouples expensive training from frequent prediction runs.

    A robust pipeline typically consists of these stages:

    1. Cleaning and Source Shaping: Transform raw datasets into predictable, cleaned tables with consistent types. This stage is dataset-specific to handle unique quirks.
    2. Standardisation: Create comparable identifiers across different sources (e.g., deriving dob_std from various raw date formats or creating phonetic encodings like surname_dm using Double Metaphone).
    3. Model Build and Training: Estimate model parameters and persist a versioned model.json artefact. Training should be treated as a separate, less frequent stage than prediction.
    4. Prediction and Edge Generation: Apply the model to generate candidate pairs (subject to blocking rules) and compute match weights/probabilities.
    5. Clustering and Entity Assignment: Group scored edges into clusters (connected components) to represent unique entities and assign stable IDs.
    6. Publishing: Export stable lookup tables for downstream consumption.

  7. Design principles for scalable Splink pipelines

    master

    To ensure linkage is repeatable and explainable at scale, follow these five design principles:

    • Modularity: Design stages and code with clear inputs and outputs. This allows you to swap datasets, rules, or output formats without rewriting the entire orchestration logic.
    • Idempotency: Ensure the same inputs and configuration always produce the same outputs. Persist artefacts to prove exactly what was run.
    • Observability: Generate charts and summary metrics (e.g., match weights, candidate volumes) as part of every run to catch shifts in data quality early.
    • Cost control: Treat candidate explosion and expensive reruns as first-class concerns. Design stages to avoid unexpected compute spikes.
    • Safe defaults: If upstream data changes, prefer failing the pipeline loudly or linking conservatively rather than allowing silent over-linking.

  8. Understand Splink link types

    master

    Splink requires a link_type to define the scope of the record comparison. The choice of link_type affects the candidate space generated, the signals used for training, and the debuggability of the output. The available types are:

    • dedupe_only: Used for finding duplicate records within a single dataset.
    • link_only: Used for finding matches between different datasets.
    • link_and_dedupe: A combination of both, finding duplicates within datasets and links between them.

    In production environments, it is often recommended to start by building dedupe_only models for each dataset to learn dataset-specific quirks before introducing cross-dataset linking.

  9. How caching works in Splink

    master

    Splink uses Caching to avoid recomputing intermediate results that are reused across a session.

    The Caching Mechanism

    1. Hashing: When a SQL pipeline is generated, Splink hashes the SQL string to create a unique identifier.
    2. Physical vs. Templated Names:
      • A templated_name is a descriptive name (e.g., __splink__df_predict).
      • A physical_name is the actual materialized table name in your database, which includes the hash (e.g., __splink__df_predict_cbc9833).
    3. Lookup: Before executing a SQL string, Splink checks if a table with that specific physical_name already exists in the database. If it exists, Splink returns the existing table instead of re-running the computation.

    Global vs. Hashed Tables

    • Hashed Tables: Most tables (like blocking results) are appended with a hash because the same logical step (e.g., __splink__df_blocked) might produce different results depending on the context. This prevents incorrect reuse.
    • Global Tables: Some tables are globally unique and do not require a hash. For example, __splink__df_concat_with_tf (the concatenation of input dataframes) is often materialized without a hash so it can be reliably found in the cache by any subsequent pipeline.

  10. Manage model artefacts and training lifecycle

    master

    In production, treat model training as a separate, versioned stage from prediction. This allows you to:

    • Decouple the high computational cost of training from weekly prediction runs.
    • Use a stable model for weekly outputs, making changes easier to interpret.
    • Version model artefacts (e.g., model.json) alongside run metadata (inputs, config, code version).

    When to retrain: Do not retrain models every week. Retrain only when justified, such as once a year or when there are material changes in the input data. The Fellegi-Sunter model is generally tolerant of small amounts of data drift.

    Monitoring: Persist Splink's exploratory, blocking, training, prediction, and clustering charts alongside each run to detect drift in identifier completeness, candidate volumes, or match weight distributions.

  11. Cluster scored edges into entities

    master

    Once scored edges are generated, the next step is clustering. Conceptually, this involves finding connected components in a graph where nodes are records and edges are the scored links.

    Why separate clustering from prediction?

    • It makes failure and recovery cleaner.
    • It is essential when combining edges from multiple models across multiple datasets.

    Operational Check: Monitor the cluster size distribution over time. Unexpectedly large clusters or shifts in cluster structure can indicate upstream data quality issues.

  12. Optimize comparison complexity

    master

    The computational intensity of your comparisons directly impacts runtime. You can optimize performance by managing the following factors:

    • Comparison Levels: Reducing the number of levels within a comparison.
    • Comparison Functions: Choosing less computationally expensive fuzzy matching functions.
    • Term Frequency Adjustments: Minimizing the use of Term Frequency Adjustments if performance is a priority.