PIICatcher

PIICatcher uses a catalog to store its state. For production, use a persistent catalog; for testing, a SQLite memory database is sufficient. Use scan_database to perform the scan on a registered source.

from dbcat.api import open_catalog, add_postgresql_source
from piicatcher.api import scan_database

# Initialize catalog (using :memory: for testing)
catalog = open_catalog(app_dir='/tmp/.config/piicatcher', path=':memory:', secret='my_secret')

with catalog.managed_session:
    # Add a source (e.g., PostgreSQL)
    source = add_postgresql_source(
        catalog=catalog, 
        name="pg_db", 
        uri="127.0.0.1", 
        username="piiuser",
        password="p11secret", 
        database="piidb"
    )
    # Run the scan
    output = scan_database(catalog=catalog, source=source)

print(output)

from dbcat.api import open_catalog, add_postgresql_source
from piicatcher.api import scan_database

catalog = open_catalog(app_dir='/tmp/.config/piicatcher', path=':memory:', secret='my_secret')

with catalog.managed_session:
    source = add_postgresql_source(catalog=catalog, name="pg_db", uri="127.0.0.1", username="piiuser",
                                    password="p11secret", database="piidb")
    output = scan_database(catalog=catalog, source=source)

print(output)

Use data_scan to identify PII by inspecting the actual values (data) within columns. This is a deeper scan than metadata_scan. It iterates through a generator of column values and applies DatumDetector objects. When a match is found, the column is labeled in the catalog, and the finding is logged to both scan_logger and data_logger.

Parameters:

• catalog: The dbcat.catalog.Catalog instance to update.
• detectors: A list of DatumDetector implementations.
• work_generator: A generator yielding Tuple[CatSchema, CatTable, CatColumn] used to calculate the total number of text columns for progress tracking.
• generator: A generator yielding Tuple[CatSchema, CatTable, CatColumn, str] containing the column metadata and the string value to scan.
• sample_size: (Optional) The number of samples to take per column. Defaults to SMALL_TABLE_MAX.

from piicatcher.scanner import data_scan
from piicatcher.detectors import DatumRegexDetector

# Assuming catalog, work_gen, and data_gen are already defined
data_scan(
    catalog=catalog,
    detectors=[DatumRegexDetector()],
    work_generator=work_gen,
    generator=data_gen,
    sample_size=100
)

Use output_dict() to transform PII findings from a Catalog and CatSource into a nested dictionary structure. The hierarchy follows source -> schemata -> tables -> columns.

By default, the function only includes columns where PII was actually detected (column.pii_type is not None). If you want to include all columns regardless of whether PII was found, set list_all=True.

Dictionary Structure:

• source: {"name": str, "schemata": [...]}
• schema: {"name": str, "tables": [...]}
• table: {"name": str, "columns": [...]}
• column: {"name": str, "data_type": str, "sort_order": int, "pii_type": str | None, "pii_plugin": str | None}

from piicatcher.output import output_dict

# Assuming catalog and source are already initialized
pii_report = output_dict(
    catalog=catalog,
    source=source,
    list_all=False,
    include_schema_regex=["prod_.*"],
    exclude_table_regex=["temp_.*"]
)

The scan_database function is the primary entry point for scanning a database source for PII (Personally Identifiable Information). It supports both metadata-level scanning (checking column names/types) and data-level scanning (inspecting actual row values).

Parameters

• catalog (Catalog): The database catalog instance.
• source (CatSource): The specific data source to scan.
• scan_type (ScanTypeEnum):
  • ScanTypeEnum.metadata: Scans metadata only (default).
  • ScanTypeEnum.data: Scans both metadata and actual data values.
• incremental (bool): If True, only scans changes since the last successful run (default: True).
• output_format (OutputFormat):
  • OutputFormat.tabular: Returns results in a tabular format (default).
  • OutputFormat.json: Returns results as a dictionary/JSON.
• list_all (bool): If True, lists all findings; otherwise, may only show new findings (default: False).
• sample_size (int): The number of rows to sample when performing a data scan (default: SMALL_TABLE_MAX).
• include_schema_regex, exclude_schema_regex, include_table_regex, exclude_table_regex (List[str]): Regex patterns to filter which schemas or tables are included/excluded from the scan.

Returns

Returns a List[Any] or Dict[Any, Any] depending on the output_format requested.

from piicatcher.api import scan_database, ScanTypeEnum, OutputFormat

# Example: Perform a full data scan on a source and get JSON output
results = scan_database(
    catalog=my_catalog,
    source=my_source,
    scan_type=ScanTypeEnum.data,
    output_format=OutputFormat.json,
    sample_size=1000
)

The piicatcher package exports several classes representing different types of Personally Identifiable Information (PII). These classes inherit from dbcat.catalog.pii_types.PiiType and are used to categorize detected sensitive data. Each class defines a human-readable name and a machine-readable type string.

Available PII types include:

• Phone (type: phone)
• Email (type: email)
• CreditCard (type: credit_card)
• Address (type: address)
• Person (type: person)
• BirthDate (type: birth_date)
• Gender (type: gender)
• Nationality (type: nationality)
• SSN (type: ssn)
• ZipCode (type: zip_code)
• PoBox (type: po_box)
• UserName (type: user_name)
• Password (type: password)

from piicatcher import Phone, Email, CreditCard, SSN

# These classes are typically used as identifiers or type markers
# within the piicatcher ecosystem to label detected data.
print(Phone.name)      # Output: Phone
print(Phone.type)      # Output: phone
print(SSN.type)        # Output: ssn
print(CreditCard.name) # Output: Credit Card