DOCSEARCHbyAlgolia
Submit Docs

Octopize Documentation

website·Indexed 33 days ago

https://docs.octopize.io

Documentation for Octopize and its Avatar platform, a privacy-preserving solution for data anonymization and synthesis. Includes guides on differentially private synthesis methods (AvatarOpenDP and AvatarFastDP), data processors, and deployment options for SaaS, Docker, and Kubernetes via Helm charts.

Tokens
53.8K
Snippets
122
Records
383
Agent score
60%

What's inside Octopize

  1. Overview of the Avatar Method for data anonymization

    The Avatar Method transforms sensitive personal data into synthetic, anonymized datasets called 'Avatars'. These datasets maintain the exact structure, granularity, and statistical properties of the original data, allowing developers and analysts to use them for secondary purposes without violating privacy regulations like GDPR or the AI Act. Because the resulting data is no longer classified as personal data, it can be used in AI models, analytics, and BI tools without compromising individual privacy.

  2. Overview of the Avatars Anonymization Pipeline

    The Avatars anonymization process is divided into three primary stages to ensure data privacy and utility:

    1. Prepare: Focuses on data formatting, preprocessing rules (file formats, data types, structure), and handling special cases like relational databases or time series to improve output quality.
    2. Run: Involves configuring anonymization parameters using the Advisor tool and executing the process via either the Web Client (for experimentation) or the Python Client (for ETL integration and advanced control).
    3. Enhance: Focuses on post-process evaluation, reviewing technical and impact assessment reports, and exporting the final synthetic dataset.

  3. Overview of Avatar Data Augmentation

    Avatar data augmentation leverages the stochastic properties of its anonymization method to generate synthetic individuals that preserve the statistical characteristics (distributions, correlations, and multivariate relationships) of the original dataset. It ensures that generated values remain within original data boundaries and inherit the same privacy guarantees as standard avatarization. This capability is compatible with tabular, multi-table, and time series methodologies.

  4. Overview of Octopize synthesis methods

    Octopize provides three synthesis methods for creating synthetic data. All three share a common four-step preprocessing pipeline: 1. Load & type, 2. Project (into latent space via FAMD), 3. Synthesize (the step that differs by method), and 4. Back-project.

    Common parameters for all methods include ncp, imputation, use_categorical_reduction, exclude_variable_names, and column_weights.

    Method Comparison:

    • Avatar (Standard): Best for general use and time series. Uses KNN in latent space. Controlled by parameter k.
    • AvatarOpenDP: Provides formal $\epsilon$-differential privacy ($\epsilon$-DP) on the synthesis step. Based on SmartNoise MST synthesizer. Slower runtime. Does not support time series.
    • AvatarFastDP: Fast, DP-flavored synthesis. Not end-to-end DP. Uses Gaussian/GMM/Histogram in latent space. Does not support time series.

    Constraint: These methods are mutually exclusive. You must set exactly one of k, open_dp_epsilon, or fast_dp_epsilon. Setting more than one will raise an error.

  5. Handle date and timestamp columns with the Datetime Processor

    The Datetime Processor automatically manages temporal variables to ensure they can be used for projections through a three-step process:

    1. Preprocess: Converts dates and timestamps into Unix timestamps (seconds since epoch).
    2. Anonymize: Performs anonymization using the numeric timestamps.
    3. Postprocess: Restores the timestamps back to the original YYYY-MM-DD format.

  6. Understand the Distance To Closest privacy metric

    The Distance To Closest metric measures the median Euclidean distance between an individual in the original dataset and the closest synthetic avatar. It evaluates how close synthetic data lies to the original distribution without relying on direct one-to-one links.

    To ensure interpretability, Octopize uses a reference-based approach comparing three sets of data:

    • WO (Working Half Original): The part of the original data being anonymized.
    • WA (Working Avatars): The synthetic avatars generated from WO.
    • HO (Holdout Half Original): A separate, untouched sample of real records used as a baseline for natural variability.

    For datasets with $\ge 500$ rows, a real holdout is used; smaller datasets use an approximate holdout created by random splitting during metric computation.

  7. Send a welcome or recovery email to a new user

    Since manually created users do not have passwords, you must send a recovery link to allow them to set one.

    1. Navigate to the user's profile page from the group's Users list.
    2. In the Overview tab, find the Recovery section and click Email recovery link.
    3. In the Email stage dropdown, you must select avatar-recovery-email to send the correct template.

  8. Evaluate synthetic data for privacy and utility

    Synthetic anonymous data generation requires a balance between replicating the characteristics of original training data (utility) and protecting the privacy of individuals. To ensure optimal quality, synthetic datasets must be quantified using objective metrics across two primary dimensions: Privacy and Utility. These metrics verify that the data retains analytical value comparable to the original dataset while meeting privacy protection criteria.

  10. Understand Personal Information under PIPEDA

    Under Part 1, Section 2(1) of the Personal Information Protection and Electronic Documents Act (PIPEDA), Personal Information is defined as "Information about an identifiable individual." The Office of the Privacy Commissioner of Canada (OPC) interprets this to mean there is a serious possibility that an individual could be identified using the information, either alone or in combination with other data.

  11. Choose an Octopize Avatar deployment method

    Octopize Avatar software can be deployed using one of two methods: as a SaaS (Software-as-a-Service) hosted on Octopize infrastructure, or self-hosted on your own infrastructure using Docker or Kubernetes. Regardless of the deployment method chosen, users have access to both the web client and the Python client.

  12. Learn the Avatar anonymization method

    To independently master the Avatar anonymization method, follow this structured learning sequence:

    1. General Introduction: Start with the Overview and keep the FAQ handy.
    2. Legal Context: Review the Legal & Compliance Overview to understand data protection laws and compliance frameworks.
    3. Technical Explanation (Optional): Explore the inner workings of the tool via the 'How the Method Works' documentation.
    4. Simple Use Case: Follow the Quickstart Tutorial for a walkthrough of the anonymization pipeline on a basic dataset.
    5. Evaluating Data Quality: Learn to measure the success of anonymization using Privacy and Utility metrics.
    6. Theoretical Application: Understand the conceptual use cases for when and why to use Avatar.
    7. Practical Application: Use the Step-by-Step User Guide to apply the method to your own data.