LaminDB

LaminDB is a distributed data management tool designed for traceable, multimodal AI workflows. It provides lakehouse guarantees (ACID transactions, time travel, schema evolution) for non-tabular data formats like parquet, zarr, AnnData, and images.

Key capabilities include:

• Lineage: Trace results across agent sessions, notebooks, scripts, and workflows.
• Lakehouse: Manage multimodal datasets with ACID guarantees and query them using engines like Polars or DuckDB.
• LIMS & ELN: Unified schema-based records management with support for ontologies and notes.
• FAIR datasets: Validate and annotate files, DataFrame, AnnData, and SpatialData.
• Governance: Manage changes via branching and versioning of both data and code.

LaminDB automatically tracks and annotates artifacts with lineage information when using tracking utilities. When you use ln.track() or the @ln.flow decorator, the resulting artifacts are automatically linked to:

• ln.Run objects (representing the execution)
• ln.Transform objects (representing the code/logic applied)

You can also pass a project argument to ln.track() to ensure all objects created during that run are automatically annotated with that project.

LaminDB provides mechanisms to support non-destructive experimentation and repeatable execution:

Branching (Write-Audit-Publish)

LaminDB uses database-level branching (stage, review, merge). Isolation is achieved via the branch field in SQLRecord.

• Isolated objects: Newly created objects live on their creation branch.
• Versioned objects: Changes to artifacts, collections, transforms, and blocks/notes create new versions isolated to the current branch.
• Note: In-place updates to existing objects are shared across all branches; to isolate such changes, one must create new objects or new database instances.

Idempotent Execution

LaminDB validates content hashes and metadata before writing. This prevents duplicate artifacts or dangling storage objects when re-running scripts, workflows (Nextflow, Snakemake), or agent traces.

Atomicity is guaranteed at the individual artifact level.

When passing a list of objects to ln.save():

• If an upload fails mid-way through a list, all successful uploads up to that point are maintained, and a RuntimeError is raised.
• If a failure occurs before any uploads begin (e.g., due to a validation error), nothing is saved to the database or storage.

# If one item in the list fails during upload, previous successful ones remain.
# If a validation error occurs before any upload, nothing is saved.
artifacts = [artifact1, artifact2, "invalid_data"]

try:
    ln.save(artifacts)
except Exception as error:
    print(f"Failed: {error}")

To prevent dangling metadata or orphaned files, LaminDB executes a save operation in three distinct phases:

1. Phase 1: Metadata is written to the database with an internal flag _storage_ongoing = True.
2. Phase 2: The actual file bytes are uploaded to the storage backend.
3. Phase 3: The database is updated to set _storage_ongoing = False.

If an upload is interrupted during Phase 2, the artifact remains flagged with _storage_ongoing = True. This state is visible in the UI. To recover, you can re-run lamin save or artifact.save() to attempt the upload again.

LaminDB provides idempotency for SQLRecord (and Record) objects through name-based lookups. When you instantiate a record with a name that exactly matches an existing record in the registry, the constructor returns the existing object instead of creating a new one.

If the name is similar to existing records, LaminDB provides search results to help you decide whether to save a new entry or use an existing one.

To disable this name-searching behavior (e.g., for performance reasons), set ln.settings.creation.search_names to False.

import lamindb as ln

# Enable name searching (default)
ln.settings.creation.search_names = True

# Create or retrieve a record by name
label = ln.Record(name="My label 1").save()

# If you want to bypass name checks for performance
ln.settings.creation.search_names = False
label = ln.Record(name="My label 1c")

When calling Collection.append(), LaminDB uses a versioning pattern similar to Iceberg or DuckLake to provide ACID guarantees and snapshot isolation without mutating existing files:

1. Artifact creation: New data is saved as a new Artifact using the two-phase commit process (staged in storage).
2. Collection versioning: A new version of the Collection is created that links to the new artifact alongside existing ones.

This provides:

• Atomicity: The new collection version is only created if the new artifact is successfully stored and registered.
• Isolation (Snapshot/Time Travel): Concurrent readers querying the original version are unaffected. The previous state remains addressable via its original version.

The Feature registry indexes variables across datasets. You can use Feature objects directly in .filter() calls for Artifact, Record, or Run registries to query by dimensions (e.g., perturbation, temperature).

feature_type = db.Feature.get(name="mini_immuno")
perturbation = db.Feature.get(name="perturbation", type=feature_type)
temperature = db.Feature.get(name="temperature")

# Filter artifacts by feature values
db.Artifact.filter(
    perturbation == "DMSO",
    temperature > 21,
).to_dataframe(include="features")

# Check if a feature is present (not null)
db.Artifact.filter(perturbation.is_null(False)).to_dataframe(include="features")