ethnicolr 2.0.0

The ethnicolr project utilizes surname frequency data from the US Census Bureau for the 2000, 2010, and 2020 census cycles. This data is used to provide demographic percentages associated with specific surnames.

Data Characteristics

• 2000 and 2010 Census: These datasets use percentages. For names with a count of 1–4, the Census Bureau suppresses the count with '(S)'. In the processed ethnicolr files, these '(S)' values are handled by equally dividing the remaining percentage across all suppressed entries.
• 2020 Census: This dataset provides raw counts instead of percentages and does not use suppression. Percentages are calculated during the ethnicolr processing stage.

The scripts/ directory contains development tools used for data acquisition and model training. These scripts are not part of the main ethnicolr package API and are intended for developers or researchers looking to:

1. Data Acquisition: Download and process raw data from sources like the Census, voter registration records (Florida and North Carolina), or Wikipedia.
2. Model Training: Train new LSTM models when new data becomes available.
3. Research: Explore new data sources and model architectures.

Note that while the scripts reside here, the resulting processed data and trained models are stored within the main package at ethnicolr/data/ and ethnicolr/models/ respectively.

Estimators like lookup_census_first_name, estimate_census_full_name, and estimate_voter_file_full_name are dictionary estimators. They use exact conditional frequencies from public tables (e.g., Census 2020) rather than neural networks.

When combining first and last names, the package uses a naive Bayes approximation:

p(race | first, last) ∝ p(race | last) · p(race | first) / π(race)

Assumption: This assumes first and last names are conditionally independent given race. For culturally correlated pairs (e.g., an Irish first name with an Irish surname), this can result in somewhat overconfident estimates. The evidence_basis column in the output tracks which evidence was used for each row.

The uncertainty_level= argument produces Monte Carlo dropout summaries. These are not confidence intervals and carry no frequentist coverage guarantee. Instead, they answer: how stable is the probability estimate itself under model uncertainty?

When using this, the following columns are added to your output:

• *_mc_mean
• *_mc_std
• *_mc_lower
• *_mc_upper

These complement conformal sets; a probability can be precisely estimated yet wrong, or noisy yet calibrated.

Ethnicolr outputs name-pattern estimates following inference contract version 1.0. This contract provides a standardized way to interpret results across different name-inference packages (like Pranaam, Instate, Outkast, and Naampy).

Key Distinction: Scoring vs. Labeling

The contract distinguishes between whether a model produced a probability distribution (scored) and whether it chose to return a specific label (abstained).

• A model can score a name (providing a probability distribution) but still abstain from providing a label if the evidence is ambiguous or meets certain uncertainty rules.
• If a model abstains, the predicted_label will be missing, but the underlying probability distribution may still be available if the row is scored.

Ethnicolr provides columns for statistical uncertainty depending on the method used:

Statistical Intervals

For standard statistical interval columns, the format is:

• <probability_column>_lower
• <probability_column>_upper

Monte Carlo Dropout Summaries

Because empirical quantiles from Monte Carlo dropout are not automatically confidence intervals, these use specific suffixes:

• <probability_column>_mc_mean
• <probability_column>_mc_std
• <probability_column>_mc_lower
• <probability_column>_mc_upper

Ethnicolr supports three distinct ways to handle uncertainty:

1. Wilson bounds: Quantifies sampling uncertainty in published Census lookup proportions.
2. Monte Carlo dropout summaries: Describes variation under repeated stochastic model passes (note: these are not confidence intervals). Use uncertainty_level and mc_iterations to trigger this.
3. Conformal prediction sets: Targets marginal coverage for names exchangeable with a model's calibration data. Use conformal_coverage for this.

The Rosenman-Olivella-Imai dictionaries provide $P(\text{race} | \text{name})$ probabilities for first and last names, derived from six Southern-state voter files (AL, FL, GA, LA, NC, SC). These dictionaries are used by the ethnicolr.estimate_voter_file_full_name function to estimate race from names.

Key components of this data include:

• first_name_race.parquet: Contains 135,777 first names with 5 race probabilities each.
• last_name_race.parquet: Contains 338,169 surnames with 5 race probabilities each.
• rosenman_stats.json: Contains provenance information and the implied voter-population race marginal (recovered via Bayes' rule), which is used as the target_prior= adjustment in estimation functions.

# Example usage context (the dictionary is used by this function)
ethnicolr.estimate_voter_file_full_name(df, first_name_col, last_name_col)

When interpreting model performance, note that metrics are not directly comparable across different data sources because they represent different populations:

• Census evaluation: Person-weighted.
• Florida evaluation: Represents registered voters.
• North Carolina evaluation: Restores pre-deduplication voter frequency.
• Wikipedia/Wikidata evaluation: Represents unique notable people.