Datahike Documentation

Datahike includes several advanced capabilities for production-grade systems:

• Query Engine: Features fused scan execution, ORDER BY support, query result caching, and d/explain (Beta).
• Secondary Indices: Supports full-text search, vector similarity, and columnar aggregates (Experimental).
• Distributed Architecture: Uses Distributed Index Space and real-time sync via Kabel.
• Versioning: Git-like branching and merging (Beta).
• Norms: A dedicated database migration system.
• Graph Algorithms: Reachability, paths, centrality, and more over a GraphSpec (Experimental).
• Optimistic Overlay: Zero-latency UI updates over a remote writer via d/with (Experimental).

Datahike is cross-platform and provides various bindings (some in beta):

• JVM: Clojure, ClojureScript, and Java APIs.
• JavaScript: Node.js and Browser (via IndexedDB).
• Python: High-level Pythonic API with automatic EDN conversion.
• C/C++: libdatahike native bindings for non-JVM applications.
• CLI: Native dthk tool (GraalVM native-image) for quick queries and automation.
• Babashka: Native-compiled pod for shell scripting.

Proximum is a high-performance HNSW (Hierarchical Navigable Small World) vector index designed specifically for Datahike's persistent data model. It enables semantic search and Retrieval-Augmented Generation (RAG) capabilities while adhering to Datahike's core principles of immutability and full audit history.

Key features include:

• Fast HNSW vector search.
• Immutable index snapshots using git-like semantics.
• Persistent data structures that operate without mutation or locks.
• Dual-licensing: EPL-2.0 (open source) and commercial.

Note: Integration as a secondary index directly into Datahike is planned for a future release.

The Java examples demonstrate three primary usage patterns:

1. Basic CRUD and Queries (QuickStart.java): Uses the builder pattern for database configuration, connects to databases, transacts data using Java Maps, and executes queries using Datalog.
2. Schema Definition and Validation (SchemaExample.java): Demonstrates defining attributes with type constraints, setting unique constraints (identity vs value), defining reference types for relationships, and managing cardinality (one vs many) using Keywords constants.
3. Historical Queries and Time Travel (TimeTravelExample.java): Leverages immutability to query the database state at specific points in time using asOf for point-in-time queries, since for change tracking, or querying the full history via transaction metadata.

Datahike's graph algorithms do not read the database directly. Instead, they operate on a GraphSpec, which is a protocol defining how to read adjacency, edges, nodes, and weights from the database. This decoupling allows algorithms to work regardless of how your edges are modeled.

Every algorithm follows the pattern:

(algorithm graph-spec db & args)

You build a spec, optionally transform it, and pass it to the algorithm.

(require '[datahike.api :as d]
         '[datahike.experimental.graph-spec :as gs]
         '[datahike.experimental.graph :as graph])

It is critical to distinguish between these two operations, especially for privacy compliance (GDPR, HIPAA, CCPA):

1. Garbage Collection (d/gc-storage): Reclaims storage by deleting old snapshots that no branch head points to. It is a routine maintenance task.
2. Data Purging: Rewrites indices to remove specific data (datoms). This creates a new commit where the indices no longer reach the targeted data.

The Erasure Recipe: To physically erase data for compliance, you must perform purge + cutoff-GC.

• A purge alone leaves the old data in the pre-purge commit (which is still a live intermediate commit).
• You must then run d/gc-storage with a grace-period cutoff old enough to drop that pre-purge commit to physically evict the nodes from storage.

Datahike storage is built on top of konserve, a universal key-value store abstraction. To add a new storage backend, you must implement a konserve backend. Once your backend is required and registered in your project, Datahike will work with it transparently.

Implementation Steps:

1. Implement a konserve backend following the konserve documentation.
2. Register your backend by requiring it in your project.
3. Use it with Datahike by specifying the backend keyword in your configuration.

The :schema-flexibility option controls when schema validation occurs:

• :write (default): Strict schema. Attributes must be defined before they can be used. This catches errors early during transactions.
• :read: Schema-less. Data is accepted without prior definition and validated only during reads. This is useful for evolving data models.

Note: If you use :attribute-refs? true, you must use :schema-flexibility :write.

{:schema-flexibility :read}  ;; Allow any data structure

Every transaction includes a meta entity that stores the current point in time in the :db/txInstant attribute. You can query this attribute to find transaction dates or join it with other entities to find when a specific piece of data was last updated.

Note that :db/txInstant is strictly monotonic. Even if multiple transactions occur in the same millisecond, the allocator ensures they are ordered by advancing the timestamp by 1ms if necessary. This ensures that an instant resolves to exactly one unique snapshot.