Presidio Research

A single model prediction may correspond to multiple distinct entities annotated in a dataset. At coarse depths, these merge effectively, but at fine depths, the model's predictions may be considered imprecise. Examples include:

• Model LOCATION $\rightarrow$ Dataset CITY, STATE, COUNTRY, POSTAL_CODE, etc.
• Model PERSON $\rightarrow$ Dataset FIRST_NAME, LAST_NAME, FULL_NAME, etc.
• Model ID $\rightarrow$ Dataset SSN, PASSPORT, DRIVER_LICENSE, etc.

The data generation process creates synthetic sentences based on templates (e.g., My name is {{name}}) and samples fake PII values.

Once generated, the process:

1. Tokenizes the data.
2. Creates tags (IO, BIO, or BILUO).
3. Creates spans for the newly created samples.

You can split this generated data into train/test/validation sets while ensuring that each template only exists in one set to avoid leakage.

A manual mapping override that is valid at a coarse depth (e.g., depth 2) may become ambiguous or incorrect at a finer depth (e.g., depth 3).

For example, mapping NRP to LOCATION at depth 2 might work for a high-level view, but at depth 3, NRP canonically resolves to NATIONALITY (under DEMOGRAPHIC). Forcing it to LOCATION creates ambiguity regarding whether it should map to LOCATION → ADDRESS or LOCATION → GPE.

When running .analyze(), the mapper identifies several types of mapping issues. These are categorized by severity:

The redesigned presidio_evaluator follows a strict three-layer architecture where each layer communicates via a standardized DataFrame. This separation of concerns ensures that the model handles prediction, the mapper handles canonicalization, and the evaluator handles scoring without overlapping responsibilities.

The Three-Layer Pipeline:

1. Model: Performs prediction using BaseModel.predict_dataset(dataset).
2. Mapper: Canonicalizes labels using CanonicalMapper.
3. Evaluator: Calculates scores using SpanEvaluator or TokenEvaluator.

Standard DataFrame Schema: All layers exchange data using a DataFrame with exactly these 5 columns:

• sentence_id (int)
• token (str)
• annotation (str)
• prediction (str)
• start_indices (int)

The presidio_evaluator uses a hierarchical approach to evaluate PII detection accuracy by projecting labels into different levels of granularity. This allows you to measure performance at different depths of the entity hierarchy.

1. Binary Level: The simplest check. Did the model detect any PII? (Labels are either PII or O).
2. Branch Level: Checks if the model identified the correct high-level category (e.g., PERSON vs LOCATION). This uses depth-2 ancestors.
3. Detailed Level: The most granular check. Compares the specific hierarchy node (e.g., NAME vs ADDRESS) identified by the model against the ground truth.

By using MappedResults, you can switch between these levels by passing the corresponding DataFrame (.binary, .branch, or .detailed) to evaluation functions or plotting tools.

For datasets using country-prefixed labels (e.g., GERMANY_PASSPORT_NUMBER or BRAZIL_CPF), the mapper uses two identification tiers before attempting fuzzy matching:

1. COUNTRY tier: Attempts to strip the country prefix (demonym or ISO code) and perform an alias lookup on the remainder.
2. COUNTRY_FALLBACK tier: Strips the leading country code and tries again.

Ensure that your labels follow consistent patterns (e.g., using ISO codes vs demonyms) to ensure reliable resolution to canonical nodes like PASSPORT or TAX_ID.

Under HIPAA, Protected Health Information (PHI) includes 18 specific types of identifiers. When building or testing de-identification systems, ensure your models or evaluators account for these categories:

• Names or part of names
• Geographical identifiers
• Phone numbers
• Email addresses
• Medical record numbers
• Account numbers
• Vehicle license plate numbers
• Web URLs
• Fingerprints, retinal and voice prints
• Any other unique identifying characteristic
• Dates directly related to an individual
• Fax numbers
• Social Security numbers
• Health insurance beneficiary numbers
• Certificate or license numbers
• Device identifiers and serial numbers
• IP addresses
• Full face or any comparable photographic images