Greenmask Documentation

Greenmask is an open-source CLI utility designed for logical PostgreSQL database backup dumping, anonymization, synthetic data generation, and restoration. It is built using ported PostgreSQL libraries to ensure reliability and is designed to be stateless, meaning it does not require any changes to your existing database schema.

Key characteristics include:

• Statelessness: Operates as a logical dump without impacting the source schema.
• Backward Compatibility: Dumps created by Greenmask are compatible with standard PostgreSQL utilities like pg_restore.
• Type Safety: Uses database drivers for encoding/decoding to ensure data integrity and format preservation.
• Parallelism: Supports parallel execution for both dumping and restoration to increase performance.
• Cross-platform: Built in Go, making it easy to execute on any platform without dependencies.

Advanced transformers are modifiable anonymization methods in Greenmask that allow users to implement custom logic for data masking. Unlike standard transformers, these can be adjusted using custom functions to meet specific requirements.

Currently available advanced transformers include:

• Json: Used to modify JSON content using delete and set operations.
• Template: Executes a user-provided Go template and applies the resulting string to a specified column.
• TemplateRecord: Modifies entire records by executing a Go template and applying changes via the PostgreSQL driver.

Greenmask configuration is organized into six primary sections:

• common: Settings applicable to both dump and restore commands.
• log: Configuration for the logging subsystem.
• storage: Settings defining where dump data is stored.
• dump: Settings for the dump command, including pg_dump options and transformation parameters.
• restore: Settings for the restore command, including pg_restore options and restoration scripts.
• custom_transformers: Definitions for custom transformers that communicate via stdin and stdout.

Greenmask extends the traditional pg_dump directory format by introducing a storages abstraction, allowing backups to be sent to locations other than the local filesystem.

Supported storage types:

• s3: Supports any S3-compatible storage system (e.g., AWS S3, MinIO, etc.).
• directory: The standard local filesystem directory.

By default, Greenmask uses COPY for restoration. You can use the --inserts flag to use INSERT commands instead. This is slower but allows for advanced error handling:

1. Error Exclusion: When combined with an error and constraint exclusion list in your configuration, you can skip specific errors and continue the restoration.
2. Conflict Handling: Use --on-conflict-do-nothing to add ON CONFLICT DO NOTHING to statements (works for unique/exclusion constraints).
3. Identity Columns: Use --overriding-system-value to allow inserting data into GENERATED ALWAYS AS IDENTITY columns.

Warning: INSERT commands are significantly slower than COPY.

Use the --restore-in-order flag to ensure that dependent tables are not restored until the tables they depend on have been restored. This is useful when the schema already exists with foreign keys.

Note on Cycles: Greenmask cannot guarantee order if the schema contains cyclic dependencies. In such cases, you must temporarily remove the foreign key constraints to break the cycle, restore the data, and then re-add the constraints.

Greenmask is strict about numeric types in expressions. To avoid incorrect comparison results (e.g., 1 == 1.0 returning false), ensure you use the correct format:

• Integer: Write the number without a decimal point (e.g., 1, 2).
• Float: Write the number with a decimal point (e.g., 1.0, 2.0).

When using RandomPerson in dynamic mode (where gender is derived from another column), you can control how values are interpreted:

gender_mapping

A dictionary that maps values found in the source column to specific genders. The default mapping is:

{
  "Male": [
    "male",
    "M",
    "m",
    "man",
    "Man"
  ],
  "Female": [
    "female",
    "F",
    "f",
    "w",
    "woman",
    "Woman"
  ]
}

fallback_gender

Specifies the gender to use if the value in the source column is not found in the gender_mapping. This is required only when the gender parameter is used in dynamic mode. Default is Any.