DBLab Engine Documentation

Database Lab Engine (DLE) provides ultra-fast, thin PostgreSQL cloning using Copy-on-Write (CoW) technologies. It allows developers to create full-sized database environments for development, testing, QA, and staging in seconds, regardless of the original database size.

Key capabilities include:

• Fast Cloning: A 1 TiB database can be cloned in approximately 10 seconds.
• Broad Compatibility: Works with any PostgreSQL instance, including AWS RDS, GCP CloudSQL, Heroku, Digital Ocean, and self-managed instances.
• Resource Efficiency: Dozens of independent clones can run on a single machine without significantly increasing hardware costs.
• Automated Workflows: Supports API and CLI for automation, and includes a UI for manual tasks.
• Data Protection: Features automatic cleanup of unused clones and a "Deletion Protection" flag to prevent accidental removal.

DLE achieves high-speed, thin cloning by leveraging Copy-on-Write (CoW) technologies. This allows the engine to create new environments that share the underlying data blocks of the source until a change is made.

DLE supports two primary technologies for CoW:

1. ZFS (Default): DLE periodically creates new snapshots of the data directory and maintains a set of snapshots. Users can choose which snapshot to use when requesting a new clone. DLE manages the lifecycle by cleaning up old, unused snapshots.
2. LVM (Logical Volume Manager for Linux).

Because it uses CoW, the theoretical maximum number of snapshots and clones is $2^{64}$ when using ZFS.

DLE utilizes Copy-on-Write (CoW) technology to enable rapid thin cloning. It supports two primary underlying technologies:

1. ZFS (Default): DLE periodically creates snapshots of the data directory and maintains a set of snapshots. Users can choose which snapshot to use when requesting a new clone. DLE manages the lifecycle by cleaning up old, unused snapshots.
2. LVM (Logical Volume Manager for Linux): An alternative CoW technology supported for thin cloning.

Because it uses CoW, the clones are 'thin', meaning they only consume additional storage for the changes made to the data after the clone is created.

DBLab supports full libpq connection strings for source database configuration. This allows you to pass all necessary parameters (such as sslmode, connect_timeout, sslrootcert, and options) directly to pg_dump and the engine's internal pgx connections.

Supported formats include:

• URI form: postgresql://...
• Keyword/Value form: host=... port=... dbname=...

While full connection strings are supported, you can still use discrete host, port, dbname, and username fields if preferred.

Instead of providing discrete host, port, user, and database fields, you can provide a single connectionString in your configuration. This allows you to pass complex parameters like sslmode or connect_timeout directly to the underlying tools.

Precedence Rules:

• If connectionString is set, it wins. The engine derives the host, port, user, and database name from it for display and validation, ignoring any individual connection.* YAML fields.
• The connectionString is passed directly to pg_dump using the -d flag.
• Passwords are never inlined into the connection string; they are always managed separately via the Password field to ensure security.

To prevent breaking lineage or losing data, the engine uses strict predicates to determine if an entity is a "true leaf" and safe to delete.

Deletable Snapshot Criteria

A snapshot is considered a true leaf (and thus deletable) only if:

• dle:child is empty.
• dle:root is empty.
• dle:branch is empty (it is not a branch head).
• It has zero clones (verified via GetCloneNumber).
• It is not returned by getBranchHeadSnapshots().

Warning: Do not rely on HasDependentEntity for this check, as it only tracks clones and does not account for child snapshots or branches.

Deletable Branch Criteria

A branch is deletable if:

• None of its snapshots have clones.
• No child branch forks from it (verified via a dependency walk).

When using the 'Detect & preview' feature in Simple Mode, the UI generates client-side warnings based on the probed configuration:

• Generic Provider Warning: If the detected provider is Generic (or cannot be mapped to a managed service), a warning appears: "Could not detect a managed cloud provider; using the generic Postgres image. Switch to Expert mode if your source runs on a managed service and we missed it."
• Memory Detection Warning: If host memory could not be detected (MemoryProbed == false), a warning appears: "Could not detect host memory; shared_buffers is set to a 1 GB safe default. Adjust in Expert mode if your host has more RAM."
• Standard Notes: The preview always includes notes regarding the RDS-refresh-tool and shared_preload_libraries.

The sweeper process runs periodically (based on checkIntervalMinutes) and evaluates each entity (snapshot or branch) using the following logic:

1. Check Protection: If the entity is Protected, clear any delete_at values and skip deletion.
2. Check Dependents: If the entity has dependents (clones, child snapshots, or child branches), clear the delete_at value (resetting the clock) and skip deletion.
3. Schedule Deletion: If the entity has no dependents and delete_at is currently unset, set delete_at to now + retention (where retention is the configured unused minutes).
4. Execute Deletion: If the entity has no dependents and now >= delete_at, trigger the deletion via destroySnapshotByID or destroyBranchByName.