DOCSEARCHbyAlgolia
Submit Docs

Graft

website·Indexed 32 days ago

https://graft.rs

An open-source transactional storage engine designed for efficient data synchronization to the edge. Graft provides a native SQLite extension that enables snapshot isolation, point-in-time restore, and global data organization via Volumes. It supports integration with JavaScript, Python, Ruby, and Swift.

Tokens
7.7K
Snippets
33
Records
72
Agent score
62%

What's inside Graft

  1. Overview of Graft storage engine

    Graft is an open-source transactional storage engine specifically designed to enable efficient data synchronization at the edge.

  2. Overview of Graft transactional storage engine

    Graft is an open-source transactional storage engine designed for efficient data synchronization at the edge. It provides a transactional layer over object storage, allowing consistent updates to subsets of data at page granularity without requiring a specific data format or schema. It is particularly suited for offline-first mobile apps, cross-platform sync, and stateless replicas in serverless or embedded environments.

    Key technical capabilities include:

    • Lazy and Partial Replication: Clients sync only the specific data they need on demand to minimize bandwidth and compute.
    • Strong Consistency: Uses Serializable Snapshot Isolation (SSI) to ensure consistent data views.
    • Instant Read Replicas: Metadata and data are decoupled, allowing replicas to start immediately without needing to replay logs or wait for full recovery.
    • Edge Optimization: The client is lightweight, making it suitable for mobile and embedded environments.

  3. Overview of Graft's architecture compared to other SQLite engines

    Graft is a stateless system built on top of object storage designed for efficient replication to and from the edge. Unlike traditional SQLite replication tools that focus on WAL (Write-Ahead Log) streaming or consensus-based clusters, Graft uses a custom VFS (Virtual File System) to implement page-level versioning. This allows for lazy, on-demand fetching of data, partial replication, and a schema-agnostic approach that supports arbitrary SQLite extensions and custom data structures.

  4. Understand Graft Volume architecture

    A Volume in Graft is a sparse, append-only collection of fixed-size 4 KiB pages. Pages are indexed starting at 1. Because Volumes are sparse, not every PageIdx needs to be filled; the Volume automatically grows to accommodate the highest written index. This architecture allows Volumes to represent entire databases (like SQLite), data shards, or partially replicated datasets. They support lazy and partial replication, meaning clients can track a subset of Pages without downloading the entire Volume.

  5. Understand Graft's Global Consistency Model

    Graft provides Serializable Snapshot Isolation globally. All read operations are executed on an isolated snapshot of a Volume. To ensure Strict Serializability, a write transaction must be based on the latest snapshot to commit successfully.

  6. Graft High-Level Architecture Overview

    Graft is a transactional storage engine designed for lazy, partial replication to the edge. It provides strong consistency with object storage durability. The architecture consists of three primary layers:

    1. SQLite Extension (libgraft_ext): Provides the VFS and Pragma interface.
    2. Runtime (graft): Manages tags, volumes, VolumeReader/Writer, and sync operations (pull/push/fetch).
    3. Storage Layer:
      • FjallStorage (Local): Handles LSM partitions, tags, volumes, logs, and pages.
      • Remote Storage (S3/FS/Memory): Stores checkpoints, commits, and segments.

  7. Understand Graft SQLite extension capabilities

    The Graft SQLite extension is a native SQLite extension that implements a SQLite virtual file system (VFS) to intercept all database reads and writes. It allows SQLite to run in resource-constrained environments by replicating only the parts of the database that a client actually uses. It maintains the same transactional semantics as SQLite in WAL (Write-Ahead Logging) mode and provides the following features:

    • Asynchronous replication to object storage
    • Stateless lazy partial replicas for edge devices
    • Serializable snapshot isolation
    • Point-in-time restore

  8. Manually download and install the Graft SQLite extension

    The Graft SQLite extension can be used anywhere SQLite is running on a supported platform and is compiled with support for loading extensions. Download the appropriate archive for your platform and architecture from GitHub Releases, then decompress it to obtain the extension binary (libgraft_ext.dll, libgraft_ext.dylib, or libgraft_ext.so).

    Available downloads:

    • Linux x86_64: libgraft-ext-linux-x86_64.tar.gz
    • Linux aarch64: libgraft-ext-linux-aarch64.tar.gz
    • Windows x86_64: libgraft-ext-windows-x86_64.zip
    • Windows aarch64: libgraft-ext-windows-aarch64.zip
    • macOS aarch64: libgraft-ext-macos-aarch64.tar.gz

  9. Understand Graft's storage and synchronization model

    Graft is a general-purpose transactional storage engine designed for partial, efficient, and flexible synchronization of data across the edge. Unlike traditional volume copying, Graft allows data to be replicated or merged at the page level.

  10. Load and verify the Graft extension in the SQLite CLI

    To use Graft within the sqlite3 command-line interface, use the .load command followed by the path to the extension binary. Once loaded, you can open a Graft Volume using the vfs=graft parameter and verify the status using the pragma graft_status command.
    -- Load the Graft extension
.load ./libgraft_ext.so

-- Open a Graft Volume as a database
.open 'file:main?vfs=graft'

-- Verify Graft is working
pragma graft_status;

  11. Import an existing SQLite database into Graft

    You can import a standard SQLite database into a new Graft Volume using the VACUUM INTO command with a Graft URI.

    Constraint: The source database page size must be 4096 (4KB). If it is not, you must set PRAGMA page_size=4096; before running the vacuum command.

    -- Open your existing database
.open /path/to/existing.db

-- Ensure page size is 4KB
PRAGMA page_size=4096;

-- Import into a new Graft tag
VACUUM INTO 'file:mytag?vfs=graft';

  12. Install the Graft SQLite extension for Ruby

    Install the sqlite-graft gem using the RubyGems CLI to enable Graft SQLite extension capabilities in your Ruby project.
    gem install sqlite-graft