DOCSEARCHbyAlgolia
Submit Docs

Graft Documentation

repository·main·Indexed 23 days ago

https://github.com/orbitinghail/graft

Graft is a transactional storage engine optimized for edge and mobile environments, featuring lazy and partial data replication with strong consistency. It is available as a Rust crate and a SQLite extension, providing a custom VFS and a suite of pragmas for volume synchronization, snapshots, and data movement (such as graft_clone, graft_fork, and graft_switch).

Tokens
21.3K
Snippets
37
Records
137
Agent score
81%

What's inside Graft

  1. Overview of the Graft SQLite extension

    main

    The Graft SQLite extension is a native extension that implements a SQLite virtual file system (VFS). It allows for selective replication of database parts that a client actually uses, enabling SQLite to run efficiently in resource-constrained environments.

    Key features include:

    • Asynchronous replication to object storage.
    • Stateless lazy partial replicas suitable for edge computing and devices.
    • Serializable snapshot isolation for consistent reads.
    • Point-in-time restore capabilities.

    The extension intercepts all reads and writes via the VFS and provides the same transactional semantics as standard SQLite running in WAL (Write-Ahead Logging) mode.

  2. Overview of Graft

    main

    Graft is an open-source transactional storage engine optimized for efficient data synchronization at the edge. It is designed to enable applications to replicate only the specific data they need through lazy and partial replication while maintaining strong consistency.

    Key Capabilities

    • Lazy & Partial Replication: Clients sync data on demand, minimizing network bandwidth and compute costs by only fetching required data.
    • Strong Consistency: Uses Serializable Snapshot Isolation (SSI) to ensure correct and consistent data views across replicas.
    • Transactional Object Storage: Transforms object storage into a transactional system. It supports consistent updates to subsets of data at page granularity without requiring a specific data format or schema.
    • Instant Read Replicas: Decouples metadata from data, allowing replicas to spin up immediately without the need for full data replay or recovery waiting periods.
    • Edge Optimization: The client is lightweight and suitable for edge, mobile, and embedded environments.

  3. What is Optimistic Snapshot Isolation in Graft?

    main

    When using the Reset and replay strategy, a client operates under Optimistic Snapshot Isolation. In this mode, reads always observe internally consistent snapshots, but these snapshots may be discarded if the subsequent commit is rejected by the server.

    This means a client might observe a state that never exists in the global timeline. If your application logic or external state changes depend on a read performed during this period, you must perform reconciliation to ensure correctness once the reset occurs.

  5. Understand Replication and Access Interfaces

    main

    Replication and data access are handled through specific abstractions:

    • Tag: A human-readable name mapped to a VolumeId.
    • SyncPoint: Manages the synchronization state between local and remote Logs. It tracks the remote LSN (the attachment point) and an optional local watermark (the last pushed LSN) to determine if a node is ahead or behind.
    • VolumeReader: A read-only, immutable interface to a Volume at a specific snapshot. Multiple VolumeReader instances can operate in parallel without locking.
    • VolumeWriter: A write interface for transactional updates to a Volume. It provides read-your-writes semantics by operating on top of an immutable Snapshot.

  6. Understand Transaction and Synchronization

    main

    Graft manages data consistency and synchronization using the following mechanisms:

    • LSN (Log Sequence Number): A monotonic, sequentially increasing number that tracks changes to a Volume. Every transaction produces a new LSN that is strictly greater than all previous LSNs for that Volume. LSNs are gapless.
    • Commit: A transaction that advances a Volume's LSN and records the specific pages that were changed.
    • Snapshot: An immutable logical view of a Volume at a specific point in time.

  7. Understand Graft Storage Concepts

    main

    Graft's storage model is built around Volumes, Pages, and Segments:

    • Volume: A sparse data object composed of Pages. Each Volume is identified by a Volume ID and tracks both a local and a remote Log for replication.
    • Page: A fixed-length block of storage. The default size is 4 KiB (4096 bytes).
    • PageIdx: The index of a page within a volume. Indexing starts at 1.
    • PageCount: The number of logical pages in a Volume. Because Volumes are sparse, writing to a high index (e.g., PageIdx(1000)) immediately sets the PageCount to that index.
    • Segment: A file containing one or more ZStd compressed frames that hold pages. Segments are tracked via Commits and are uploaded to Remote Storage.

  10. Understand the Graft High-Level Architecture

    main

    Graft is a transactional storage engine designed for lazy, partial replication to the edge. It provides strong consistency with object storage durability. The architecture consists of three main layers:

    1. SQLite Extension (libgraft_ext): Provides a VFS and Pragma interface for integration with SQLite.
    2. Runtime (graft): Manages tags, volumes, VolumeReader/VolumeWriter instances, and synchronization operations (pull/push/fetch).
    3. Storage Layer:
      • Local (FjallStorage): Manages LSM partitions containing tags, volumes, logs, and pages.
      • Remote: Supports S3, Filesystem, or Memory, storing checkpoints, commits, and segments.

    Pages are loaded lazily on-demand. When a reader requests a page, Graft identifies the segment containing it and fetches the corresponding frame from remote storage if it is not already cached locally.

  11. Understand Graft Storage Implementation

    main

    Graft utilizes two distinct storage layers:

    • FjallStorage: The local LSM-tree based storage layer. It is responsible for storing Tags, Volumes, Logs, and Pages, acting as a fast local cache and transaction staging area.
    • Remote Storage: The shared source of truth used for replication. It can be implemented using S3, a filesystem, or memory, and is responsible for storing commits and segments.

  12. How Graft's Core Transaction Model works

    main

    Graft uses snapshot isolation combined with strict commit serialization to enable safe concurrent access.

    Lock-Free Concurrent Reads

    All reads operate against immutable Snapshots. A Snapshot is a logical view of a Volume at a specific point in time, consisting of LSN ranges from one or more logs. Because snapshots are immutable, multiple VolumeReader instances can read in parallel without locking.

    Read-Your-Write Semantics

    VolumeWriter instances provide transactional write isolation. Each writer maintains:

    • An immutable base snapshot from the transaction start.
    • A staged segment tracking all pages modified in the transaction.

    Within a transaction, reads will see uncommitted writes from that same transaction (read-your-write semantics).

    Strictly Serialized Commits

    Commits use optimistic concurrency control through these steps:

    1. Validation Phase: Verify the base snapshot is still the latest version.
    2. Serialization: Acquire a global write lock to ensure commits execute one at a time.
    3. Write: Append the new commit to the log with a monotonically increasing LSN.
    4. Conflict Detection: If validation fails, the transaction must abort and retry.