csvmatch

Blocking is an advanced technique where multiple matches are performed in sequence. Each subsequent match only applies to the subset of results from the previous match. This makes matching faster and more precise for large datasets.

To implement blocking, you provide lists of arguments (fields, ignores, methods, and thresholds). If you provide fewer values than there are blocks, the last specified value is reused for all subsequent blocks. You can use --verbose to verify how your blocks are being parsed.

# Example: Block 1 (case-insensitive surname match) -> Block 2 (Levenshtein forename match)
$ csvmatch \
    data1.csv \
    data2.csv \
    --fields1 name \
    --fields2 'Person Name' \
    --ignore case words-leading \
    --ignore words-tailing \
    --method literal levenshtein \
    > matches.csv

The recommended installation method is using pipx. If you encounter installation errors, it is likely due to an unsupported Python version. You can specify a required version using the --python flag. If the command is not found after installation, run pipx ensurepath.

# Install via pipx
pipx install csvmatch

# If you need to specify a specific Python version
pipx install --python 3.x --fetch-missing-python csvmatch

# If 'command not found' occurs after installation
pipx ensurepath

To perform a basic exact match, provide the two input files (CSV or Parquet) as arguments, followed by --fields1 and --fields2 to specify the columns to compare. Wrap column names in quotes if they contain spaces. The output is written to stdout and can be redirected to a CSV file.

$ csvmatch \
    data1.csv \
    data2.csv \
    --fields1 name \
    --fields2 'Person Name' \
    > matches.csv

The csvmatch command-line tool allows you to compare two files (CSV or Parquet) and perform fuzzy or literal matching on specific columns. It supports various join types and matching algorithms like Levenshtein or Jaro.

Basic Usage:

csvmatch [options] FILE1 FILE2

Key Capabilities:

• File Formats: Supports .csv and .parquet files.
• Encoding: Automatically detects encoding for CSV files, or you can specify it manually using --enc1 or --enc2.
• Join Types: Supports inner, left-outer, right-outer, and full-outer joins via the --join flag.
• Output Control: Use --output to select specific columns from the source files (prefixed with 1. or 2.), include all columns from a file (1* or 2*), or include the match strength (degree).

csvmatch FILE1.csv FILE2.csv

You can control how records are matched using the --method and --threshold flags.

Matching Methods (--method):

• literal: Exact string matching (default).
• levenshtein (or leven): Edit distance matching.
• jaro: Jaro similarity matching.
• metaphone: Phonetic matching.
• bilenko: Bilenko similarity matching.

Thresholds (--threshold): Specify a number between 0 and 1 representing the minimum similarity required for a match. If multiple thresholds are provided, they are applied to the corresponding methods.

Example: To use Levenshtein matching with a threshold of 0.8:

csvmatch --method levenshtein --threshold 0.8 FILE1.csv FILE2.csv

The --fields1 and --fields2 arguments define which columns from the first and second files should be used for the match.

• You can provide multiple columns separated by spaces.
• The columns must be in the same order for both files (the first column in --fields1 is compared against the first column in --fields2, etc.).
• If a column name contains spaces, wrap it in quotes.
• If omitted, all columns are compared by default.
• This flag can be specified multiple times to implement blocking.

$ csvmatch \
  data1.csv \
  data2.csv \
  --fields1 name place \
  --fields2 'Person Name' Location \
  > matches.csv

Available join types:

• inner: (Default) Only returns rows that successfully matched.
• left-outer: Returns everything from the first file; non-matches from the second file will be blank.
• right-outer: Returns everything from the second file; non-matches from the first file will be blank.
• full-outer: Returns everything from both files.

# Example: Perform a left-outer join to keep all rows from data1.csv
$ csvmatch data1.csv data2.csv --fields1 name --fields2 'Person Name' --join left-outer

The --output argument allows you to specify which columns appear in the resulting CSV. Columns are prefixed with 1. or 2. to indicate their source file.

Special column identifiers:

• 1*: All columns from the first file.
• 2*: All columns from the second file.
• degree: The matching degree number (if using a method that generates one).

# Example: Output all columns from file 2, then codename from file 1, then the match degree
$ csvmatch data1.csv data2.csv --fields1 name --fields2 'Person Name' --method levenshtein --output '2*' 1.codename degree

The --ignore flag allows you to strip certain characteristics from strings before matching. You can provide multiple ignore properties using the -i flag.

Supported Properties:

• c or case: Ignore character case.
• na or nonalpha: Ignore non-alphabetic characters.
• nl or nonlatin: Ignore non-Latin characters.
• wl or words-leading: Ignore leading whitespace/words.
• wt or words-tailing: Ignore trailing whitespace/words.
• wo or words-order: Ignore word order.
• t or titles: Ignore titles.
• r=EXPRESSION or regex=EXPRESSION: Ignore strings matching a specific regular expression.

Note: If you provide a filename after regex=, csvmatch will read the regex patterns from that file (one per line).

Example: Ignore case and non-alphabetic characters:

csvmatch -i case -i nonalpha FILE1.csv FILE2.csv