DOCSEARCHbyAlgolia
Submit Docs

Presidio - Data Protection and De-identification SDK

repository·main·Indexed 27 days ago

https://github.com/data-privacy-stack/presidio

An SDK for identifying and protecting sensitive data in structured (Pandas DataFrames) and semi-structured (JSON) formats. It includes the presidio-analyzer for PII/PHI detection using spaCy, local LLMs via Ollama, or Azure OpenAI, and the presidio-anonymizer for data redaction, masking, hashing, and encryption. Supports multiple languages, including English and German, and provides tools for deanonymization and GPU acceleration.

Tokens
103.4K
Snippets
237
Records
460
Agent score
93%

What's inside Presidio

  1. Overview of supported PII entities in Presidio

    main
    Presidio provides a set of predefined recognizers for detecting various Personally Identifiable Information (PII) entities. These entities are categorized by their scope (Global or country-specific). If the predefined entities do not meet your requirements, Presidio supports adding custom entity recognizers.

  2. Overview of Presidio components

    main

    Presidio is a suite of tools designed for detecting and de-identifying Personally Identifiable Information (PII) across different data formats. The core components include:

    • Analyzer: Detects PII in text.
    • Anonymizer: De-identifies or masks detected PII.
    • Image Redactor: Redacts PII found within images, including standard image types and DICOM medical images.
    • Structured: Handles PII detection and anonymization in structured data (e.g., DataFrames).

  3. Overview of Presidio

    main

    Presidio is a context-aware, pluggable, and customizable PII (Personally Identifiable Information) de-identification SDK designed for text and images. It provides modules for the fast identification and anonymization of sensitive entities such as credit card numbers, names, locations, social security numbers, bitcoin wallets, and more.

    Key features include:

    • Predefined or custom PII recognizers using Named Entity Recognition (NER), regular expressions, rule-based logic, and checksums.
    • Extensibility to connect to external PII detection models.
    • Versatile deployment options including Python, PySpark, Docker, and Kubernetes.
    • Image Redaction capabilities for standard image types and DICOM medical images.

  4. Overview of Presidio modules

    main

    Presidio is a suite of tools designed to identify and anonymize sensitive data (PII) in various formats. It consists of four primary modules:

    1. Presidio analyzer: Used for identifying PII within unstructured text.
    2. Presidio anonymizer: Used for de-identifying detected PII entities using various anonymization operators.
    3. Presidio image redactor: Used for detecting and redacting PII entities from images using OCR and PII identification.
    4. Presidio structured: Used for identifying PII within structured or semi-structured data (e.g., DataFrames).

  5. Understand the Presidio HTTP REST endpoint ADF pipeline activities

    main

    The solution template for the HTTP REST endpoint approach uses the following seven ADF activities to process JSON documents from GitHub, anonymize them, and store them in Azure Blob Storage:

    • GetDataSet: Copies the dataset from GitHub to the /dataset folder in Azure Storage.
    • LoadSet: Loads the dataset into ADF memory for a for-each loop.
    • GetSASToken: Retrieves the SAS token from Azure Key Vault for writing to the blob container.
    • SaveBlobs: A For-Each loop activity executed for each document in the array.
    • PresidioAnalyze: Sends text to the Presidio analyzer endpoint.
    • PresidioAnonymize: Sends the analyzer response to the Presidio anonymizer endpoint.
    • UploadBlob: Saves the anonymized response as a randomly named text file in the target Azure Blob Storage.

  6. High-level concepts in Presidio

    main

    Understanding the core terminology used in Presidio for PII detection and management:

    • Entity: A span of text that identifies an individual (e.g., email, phone number), represented by a RecognizerResult object.
    • Context: The surrounding text of an entity used to improve detection accuracy.
    • Recognizer: An object responsible for detecting entities. Can be rule-based, machine learning-based, or both. Key types include EntityRecognizer and PatternRecognizer.
    • Analyzer: The AnalyzerEngine orchestrates multiple recognizers to detect PII.
    • Recognizer Types:
      • Predefined recognizer: Built-in recognizers provided by Presidio.
      • Custom recognizer: User-added recognizers.
      • ad-hoc recognizer: A recognizer added directly to a specific request rather than the global registry.
    • Filtering Lists:
      • Deny list: Terms that should always be identified as PII.
      • Allow list: Terms that should never be identified as PII.

  7. Understand Presidio core concepts

    main

    Presidio is an open-source SDK and library designed for the identification and anonymization of sensitive/PII data in text and images. Unlike SaaS-based PII detection services (e.g., Azure AI Language, Amazon Comprehend), Presidio is highly customizable and pluggable, allowing developers to adapt it to specific business needs, custom entity types, or new languages.

    Note: Presidio is a library/SDK, not a managed service. It uses automated detection mechanisms, so it should be used as part of a broader defense-in-depth strategy.

  8. Understand the Presidio on Databricks ADF pipeline activities

    main

    The Databricks-based solution template uses the following seven ADF activities to process large datasets using Spark:

    • GetDataSet: Copies the dataset from GitHub to the /dataset folder in Azure Storage.
    • GetSASToken: Retrieves the SAS token from Azure Key Vault.
    • LoadSet: Loads the dataset into ADF memory for a for-each loop.
    • SaveBlobs: A For-Each loop activity.
    • UploadBlob: Saves the un-anonymized text file to a temporary container in Azure Blob Storage.
    • GetSecret: Retrieves the storage account secret from Azure Key Vault for Databricks access.
    • Presidio-Anonymize: A Databricks Spark job that runs Presidio on the temporary storage container. The output is saved as CSV files in the /output container.

  9. Access Presidio Python API references

    main

    Presidio provides specialized Python APIs for different privacy tasks. You can find detailed API references for the following modules:

    • Presidio analyzer: For detecting PII (Personally Identifiable Information).
    • Presidio anonymizer: For masking or replacing detected PII.
    • Presidio image redactor: For detecting and redacting PII within images.
    • Presidio structured: For performing PII analysis and anonymization on structured data like Pandas DataFrames.

  10. Enable Language Model-based PII/PHI Detection

    main

    Presidio supports experimental language model-based detection for PII (e.g., Names, SSN) and PHI (e.g., Medical records) using LLMs or SLMs. This approach uses the LangExtract library to integrate with providers like Azure OpenAI (cloud-based) or Ollama (local/on-premises).

    Unlike pattern-based recognizers, detection capability depends on the specific model, the prompt description, and configured few-shot examples. You can customize these to detect any entity relevant to your use case.

  11. Core components of Presidio Analyzer

    main

    The Presidio Analyzer is built around several key classes:

    • AnalyzerEngine: The primary entry point for detecting entities in text. It orchestrates the NlpEngine, RecognizerRegistry, and ContextAwareEnhancer.
    • RecognizerResult: A data object holding the entity_type, score, start, and end position of a detected PII entity.
    • EntityRecognizer: An object responsible for detecting specific entities. It can be rule-based, machine learning-based, or a hybrid.
    • PatternRecognizer: A specific type of EntityRecognizer that uses regular expressions, context words, and validation logic to detect entities.
    • RecognizerRegistry: A container for all available EntityRecognizer objects used by the engine.
    • NlpEngine: Handles text parsing and feature extraction (tokens, lemmas, etc.) using models like spaCy, Stanza, or HuggingFace. It supports batch processing for large volumes of text.
    • ContextAwareEnhancer: Improves detection accuracy by using surrounding text context (e.g., using the LemmaContextAwareEnhancer to look at token lemmas).

  12. Main objects in Presidio

    main

    The following classes and interfaces form the core of the Presidio ecosystem:

    Analysis & Detection

    • AnalyzerEngine: The primary class for orchestrating PII detection. It uses an NlpEngine for text processing and a RecognizerRegistry to manage recognizers.
    • RecognizerRegistry: A class that holds the various recognizers used by the AnalyzerEngine.
    • NlpEngine: An interface for text processing. Implementations include SpacyNlpEngine, TransformersNlpEngine, and StanzaNlpEngine.
    • EntityRecognizer: An object responsible for detecting entities via rules or ML models.
    • RecognizerResult: An object holding the type and span of a detected PII entity.
    • BatchAnalyzerEngine: Used for detecting PII entities across a batch of texts.

    Anonymization & Redaction

    • AnonymizerEngine: The main class for anonymizing PII entities using results from the AnalyzerEngine.
    • Operator: An object that performs the actual anonymization (e.g., Replace, Redact, Encrypt).
    • BatchAnonymizerEngine: Used for anonymizing a batch of texts.
    • DeanonymizerEngine: Used to reverse anonymization (e.g., decryption) if the operation is reversible.

    Specialized Engines

    • ImageRedactorEngine: Redacts PII in images by leveraging the AnalyzerEngine on extracted text.
    • StructuredEngine: Detects PII in structured data by using the AnalyzerEngine on text fields.