DOCSEARCHbyAlgolia
Submit Docs

DefraDB Documentation

repository·develop·Indexed 21 days ago

https://github.com/sourcenetwork/defradb

A zero-trust, multi-write-master database designed for data verifiability and privacy. DefraDB utilizes MerkleCRDTs, IPLD content-addressability, and delta-state CRDTs (including LWWRegister, PCounter, PNCounter, and OR-Map) to enable decentralized data synchronization and peer-to-peer networking. It features a multi-layer cache architecture for transaction isolation and Searchable Encryption (SE) using HMAC-SHA256 for privacy-preserving queries on encrypted fields.

Tokens
129.8K
Snippets
392
Records
624
Agent score
74%

What's inside DefraDB

  1. Overview of DefraDB core concepts

    develop

    DefraDB is a user-centric database designed for data ownership, privacy, and security. It utilizes a multi-write-master architecture enabled by the convergence of MerkleCRDTs and IPLD content-addressability.

    Key features include:

    • DQL: A query language compatible with GraphQL that provides additional convenience features.
    • P2P Networking: Enables deployment in various network topologies using peer-to-peer protocols.
    • Relationship-based Access Control: Uses a Domain Specific Language (DSL) to support document or field-level security policies, secured via the SourceHub network.
    • Decentralized Data Model: Part of the Source technologies ecosystem for decentralized data and access-control management.

  3. New features in DefraDB v0.6.0

    develop

    DefraDB v0.6.0 introduced several key capabilities:

    Schema Management

    • LensVM Schema Migration Engine: A new engine powered by WebAssembly for managing schema migrations.
    • Schema Migration CLI: New get and set commands for schema migrations.
    • Schema List API: Ability to list available schemas.

    Querying and Operators

    • _not operator: Added support for negation in queries.
    • Relation Aliasing: Support for relation aliases (e.g., type_id) during create and update operations to improve developer experience.
    • Secondary Indexes: New CLI commands for managing secondary indexes, including support for non-unique secondary indexes (though these cannot be queried).

    Developer Tools

    • DefraDB Playground: An embedded environment that includes a bundled GraphQL client and schema manager.
    • Data Import/Export: Simple mechanisms for importing and exporting data.
    • Explain-Debug: Ability to explain-debug all nodes for better query performance analysis.

  4. New features in DefraDB v0.10.0

    develop

    The v0.10.0 release introduced several new capabilities for data modeling and querying:

    • JSON Support: Added JSON scalar type and validation for JSON fields.
    • Advanced Querying: Added a case-insensitive like operator and support for composite indexes.
    • Views: Added support for views with Lens transforms.
    • Schema Management: Added support for SchemaVersions and migrations on Collections (Cols).
    • Nullability: Allowed setting null values on document fields, including support for multiple documents with nil values on unique-indexed fields.
    • Tooling: Added an OpenAPI command to generate OpenAPI specifications.
    • API Changes: Reworked GetCollection and SchemaByFoo functions into a single unified function.

  5. New features in DefraDB v0.2

    develop

    DefraDB v0.2 is a major pre-production release that introduces several core capabilities:

    • P2P Data Synchronization: A new Peer-to-Peer networking system for data sync.
    • TimeTraveling Queries: A history-traversing query engine and doc fetcher that allows querying previous states of a document.
    • Expanded Query System: Support for GroupBy and Aggregate operations, including sum and count aggregates.
    • Document Management: Ability to delete documents using a specific Key.
    • Advanced Data Types: Support for lwwr scalar arrays (full replace on update) and named relationships.
    • Lookup Capabilities: Support for multi-document key lookups.

  6. New features in DefraDB v0.11.0

    develop

    The v0.11.0 release introduces several new capabilities and architectural improvements:

    Core Features

    • Access Control Policy: New mechanism for managing permissions.
    • Defra-Lens Support: Support for branching schema via Defra-Lens.
    • P Counter CRDT: Implementation of the P Counter Conflict-free Replicated Data Type.
    • PatchCollection: New way to manage mutations of collection sources.
    • Lens Runtime Config: Configuration options for the Lens runtime.

    Schema and Data Model Changes

    • Explicit Primary Declarations: One-to-one relations now require explicit primary declarations in the SDL.
    • Relation Field Properties: Relation field properties have been moved onto the collection level.
    • Nillable Arrays: All array kinds are now nillable.

    API and Refactoring

    • Unified Collection Operations: UpdateWith and DeleteWith have been merged into a single operation.
    • Corelog Integration: The logging package has been replaced with corelog (version 0.0.7).

  7. What is the DefraDB Query Language (DQL)?

    develop

    The DefraDB Query Language (DQL) is a GraphQL-defined API used to access and query data residing within a DefraDB node. Unlike graph-specific databases like DGraph, DefraDB is designed as a document storage database.

    DQL provides direct access to all database functionalities, including:

    • Reading, writing, and modifying data.
    • Describing data structures and schemas.
    • Architecting data models using indexes and other schema-independent, application-specific requirements.

    Note that while DQL handles data operations, the PeerAPI is the specific interface used for interacting with other databases and the underlying CRDTs (Conflict-free Replicated Data Types) for collaborative text editing.

  8. What is IPFS and how does it work?

    develop

    IPFS (InterPlanetary File System) is a decentralized peer-to-peer (p2p) storage network that uses content addressing to access websites, applications, and data. Unlike traditional location-based addressing (URLs), IPFS identifies files by their cryptographic hash, ensuring data integrity and uniqueness.

    Core Principles

    1. Content Addressing: Files are identified by a unique cryptographic hash of their content rather than a specific server location.
    2. Content Linking: Data is organized and linked using Directed Acyclic Graphs (DAGs).
    3. Content Discovery: The network locates data using Distributed Hash Tables (DHTs).

    Benefits of the IPFS Model

    • Decentralization: Data is accessed from multiple locations not managed by a single organization, making it resistant to censorship.
    • Verification: The content identifier allows you to verify that the received data matches the requested content.
    • Efficiency: Enables faster file transfers by accessing data from the nearest available nodes.

  9. Understand secondary index epoch namespacing changes

    develop

    In recent updates, secondary index entries are now namespaced by an epoch. Each index tracks its live epoch within the system store.

    Critical Breaking Change: This change modifies how index entries are keyed. As a result, index data written by earlier versions of Defradb is not readable by versions implementing this epoch namespacing. If you are upgrading, you should expect to rebuild or re-index your data to ensure compatibility with the new keying scheme.

  10. DefraDB Request Entry Points

    develop

    Requests can enter DefraDB through several interfaces, all of which converge at the unified client DB interface for core logic processing:

    • Direct Go Client: For embedded usage (client/db.go).
    • HTTP API: Via http/handler.go, supporting:
      • REST endpoints
      • GraphQL endpoints
    • CLI: Command-line interface wrapper (cli/client.go).