dedupe-examples

Dedupe relies on human input (active learning) to learn the rules for deduplicating data. To achieve high-quality results, you must provide labeled examples of matches and non-matches.

Best Practices:

• Provide at least 10 positive matches and 10 negative matches.
• The training results are saved to a JSON file, which allows dedupe to perform future runs using the learned rules.

Labeling Interface Example: When prompted, you compare two records and respond with:

• y: Yes (they are the same entity)
• n: No (they are different entities)
• u: Unsure
• f: Finished (stop training)

When running mysql_example.py, the script enters an active learning phase where you must provide feedback to train the deduplication model. Use the following keys to flag records:

• y: Flag as a duplicate
• n: Flag as not a duplicate
• u: Unflag/Undo
• f: Finish the process

To run the Big PostgreSQL deduplication example, follow these steps:

1. Install Python dependencies: Use pip to install the required packages.
2. Prepare PostgreSQL: Create a new PostgreSQL database and ensure the intarray extension is installed/enabled in that database.
3. Configure Connection: Set the DATABASE_URL environment variable with your connection string.
4. Initialize and Run: Run the database initialization script followed by the main example script.

# Install dependencies
pip install -r requirements.txt

# Create database and set connection string
createdb campfin
export DATABASE_URL=postgres:///campfin

# Initialize the database schema
python pgsql_big_dedupe_example_init_db.py 

# Run the deduplication example
python pgsql_big_dedupe_example.py

The Postgres Gazetteer example requires a running PostgreSQL instance. You must create a database and provide a connection string via the DATABASE_URL environment variable before running the script.

createdb dedupe_example
export DATABASE_URL=postgres:///dedupe_example

The Gazetteer examples are split into two versions based on the dedupe API version used. You must install the corresponding requirements file in a separate environment for each.

• For the in-memory example (uses dedupe 2.x API), use requirements-2.x.txt.
• For the Postgres example (uses dedupe 1.x API), use requirements-1.x.txt.

# For in-memory (dedupe 2.x)
pip install -r requirements-2.x.txt

# For Postgres (dedupe 1.x)
pip install -r requirements-1.x.txt

To use these examples, clone the repository and set up a Python virtual environment. It is recommended to use virtualenv and virtualenvwrapper.

1. Clone the repository:

git clone https://github.com/dedupeio/dedupe-examples.git
cd dedupe-examples

2. Create and configure the environment:

mkvirtualenv dedupe-examples
pip install -r requirements.txt

3. To resume work later:

workon dedupe-examples

This example processes IL campaign contribution data to identify unique donors using a MySQL database. Note that the process can take several hours and is resource-intensive; running it overnight is recommended.

Prerequisites

1. Create a MySQL database named contributions.
2. Configure database credentials:
   • Copy mysql_example/mysql.cnf_LOCAL to mysql_example/mysql.cnf.
   • Edit mysql_example/mysql.cnf with your MySQL username and password.
3. Install required Python dependencies:

   pip install -r requirements.txt

Execution Steps

Navigate to the example directory and run the initialization script followed by the main example script:

cd mysql_example
python mysql_init_db.py 
python mysql_example.py

This example demonstrates using dedupe to disambiguate inventor data from the PATSTAT international patent dataset. It specifically showcases advanced features including:

1. Set and Class Comparisons: Useful for attributes that are collections (e.g., lists of coauthors or technical IPC codes).
2. Geographic Distance Comparison: Uses Haversine distance to compare latitude and longitude coordinates.
3. Advanced Blocking: Implements blocking predicates based on both sets and geographic distance.

To execute the disambiguation process, run the following command in your terminal:

python patent_example.py

This example processes a list of early childhood education sites in Chicago from 10 different sources using CSV data.

1. Navigate to the directory:

cd csv_example

2. Install required dependency:

pip install unidecode

3. Run the script:

python csv_example.py

During execution, you will be prompted to label records for active learning using the following keys:

• y: Yes (match)
• n: No (not a match)
• u: Unsure
• f: Finished

This example links two separate spreadsheets of electronics products. Note that in this scenario, each dataset individually contains no duplicates; the goal is to find matching entries across the two datasets.

cd record_linkage_example
python record_linkage_example.py

To evaluate the performance of the disambiguation process against the provided reference data (mapping PATSTAT identifiers to manually-checked unique identifiers), run the evaluation script:

python patent_evaluation.py

The in-memory example uses the dedupe 2.x API. Running the main script produces a matching output file named gazetteer_output.csv. You can then run an evaluation script to assess the results.

# Run the matching job
python gazetteer_example.py

# Run the evaluation
python gazetteer_evaluation.py