DOCSEARCHbyAlgolia
Submit Docs

Sync Engine Web

repository·main·Indexed 42 days ago

https://github.com/typeonce-dev/sync-engine-web

A local-first, offline-capable web sync engine using CRDTs (via loro-crdt) to synchronize data across multiple devices. It features a client-side architecture using IndexedDB (Dexie) and Web Workers, with a client-agnostic REST API server for byte storage and JWT-based authentication. The engine ensures data consistency through a local-first model where the server acts as a sync and backup layer.

Tokens
11.1K
Snippets
35
Records
43
Agent score
53%

What's inside Sync Engine Web

  1. Overview of Sync Engine Web

    main

    Sync Engine Web is a local-first, offline-capable web sync engine that uses CRDT-based synchronization to maintain data consistency across multiple devices. It is designed for reliable data synchronization where the server acts as a sync and backup layer, while all primary data operations and querying occur client-side.

    Core Architecture

    • Client: A web-based implementation using IndexedDB (via dexie) for local storage, Web Workers for background synchronization, and client-side schema migrations.
    • Server: A client-agnostic REST API that stores encoded CRDT data as bytes. It manages workspaces, backups, and JWT-based authentication.
    • Shared Packages: Contains schema definitions, type definitions, and migration utilities used by both client and server.

  2. How the Sync Engine Web architecture works

    main

    The engine follows a local-first mental model. Data flows between the local storage and the server through a background synchronization layer.

    Data Flow Model

    Client (Local Storage) <-> Web Worker (Background Sync) <-> Server (Byte Storage)

    Key Principles

    • Local-First: The UI reads directly from local storage. Data is primarily managed locally.
    • CRDT-Based Sync: Uses loro-crdt to ensure conflict-free synchronization between devices.
    • Client-Side Operations: No server-side querying is performed; all data operations happen on the client.
    • Server Role: The server is client-agnostic, meaning it only stores encoded CRDT bytes and does not need to understand the underlying data schema.

  3. How authentication and workspace access works

    main

    The system uses a master/access token model for secure multi-device access.

    Authentication Flow

    1. Workspace Creation: A client creates a local workspace.
    2. Master Designation: The server stores the workspace and designates the creating client as the "master".
    3. Master Token: The master client receives a master token.
    4. Access Token Generation: The master client generates access tokens for other devices.
    5. Sharing: Access tokens can be shared via links to allow other devices to join the workspace.
    6. Verification: The server handles JWT-based token verification and authorization for all requests.

  4. Authentication and Middleware Concepts

    main

    The API uses middleware to enforce security and data integrity:

    • ApiKey Security: Authentication is performed via the x-api-key header.
    • Authorization Middleware: Validates the token and provides the AuthWorkspace context. It can fail with Unauthorized, MissingWorkspace, or DatabaseError.
    • MasterAuthorization Middleware: A stricter version of authorization that requires a master token. Used for administrative tasks like issuing or revoking tokens.
    • VersionCheck Middleware: Ensures the client's snapshot version is compatible with the server. It provides the ValidDoc context (the Snapshot type) and can fail with a VersionError.

    Context Tags

    When middleware succeeds, it provides data via Context Tags:

    • AuthWorkspace: Provides the WorkspaceTable for the authenticated workspace.
    • ValidDoc: Provides the Snapshot type for the validated document.
    const authKey = "x-api-key";
export const ApiKey = HttpApiSecurity.apiKey({
  in: "header",
  key: authKey,
});

export class VersionCheck extends HttpApiMiddleware.Tag<VersionCheck>()(
  "VersionCheck",
  {
    failure: VersionError,
    provides: ValidDoc,
  }
) {}

export class Authorization extends HttpApiMiddleware.Tag<Authorization>()(
  "Authorization",
  {
    failure: Schema.Union(Unauthorized, MissingWorkspace, DatabaseError),
    provides: AuthWorkspace,
    security: { apiKey: ApiKey },
  }
) {}

  5. Understand the LoroSchema structure

    main

    The LoroSchema type defines the expected shape of the underlying Loro CRDT document. It consists of:

    • metadata: A LoroMap containing versioning information (e.g., version).
    • food: A LoroList of LoroMap objects, where each map follows the structure defined by the current schema's food field.
    • meal: A LoroList of LoroMap objects, where each map follows the structure defined by the current schema's meal field.

  6. Configure the pgAdmin service via Docker Compose

    main

    The pgadmin service provides a web-based interface for managing PostgreSQL. It uses environment variables from a .env file for authentication. The service is accessible on host port 5050.

    services:
  pgadmin:
    env_file: .env
    container_name: pgadmin
    image: dpage/pgadmin4:latest
    environment:
      - PGADMIN_DEFAULT_EMAIL=${PGADMIN_MAIL}
      - PGADMIN_DEFAULT_PASSWORD=${PGADMIN_PW}
    ports:
      - 5050:80

  7. Configure the PostgreSQL service via Docker Compose

    main

    The postgres service runs a PostgreSQL 16 instance. It uses environment variables from a .env file to configure the database credentials and name. The database is exposed on host port 5435.

    services:
  postgres:
    env_file: .env
    container_name: postgres
    image: postgres:16-alpine
    environment:
      - POSTGRES_USER=${POSTGRES_USER}
      - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
      - POSTGRES_DB=${POSTGRES_DATABASE}
    ports:
      - 5435:5432

  8. Configure Drizzle ORM for the server

    main

    The server uses Drizzle ORM for database management. The configuration is defined using defineConfig from drizzle-kit. The current configuration specifies the output directory for migrations, the location of the database schema, and the database dialect.

    import { defineConfig } from "drizzle-kit";

export default defineConfig({
  out: "./drizzle",
  schema: "./src/db/schema.ts",
  dialect: "postgresql",
});

  9. Configure the Server environment variables

    main

    The server requires several environment variables to connect to the PostgreSQL database and handle authentication via JSON Web Tokens (JWT). Set these variables in your .env file to configure the connection parameters and security.

    POSTGRES_PASSWORD=postgres
POSTGRES_USERNAME=postgres
POSTGRES_DATABASE=postgres
POSTGRES_HOST=localhost
POSTGRES_PORT=5435
JWT_SECRET=

  10. Technical Stack and Dependencies

    main

    The project is built using the following technology stack:

    Client-Side

    • UI & Routing: react, @tanstack/react-router
    • Storage & Sync: loro-crdt (CRDT implementation), dexie (IndexedDB wrapper), dexie-react-hooks
    • Core: effect, @effect/platform-browser

    Server-Side

    • Database: drizzle-orm, @effect/sql, @effect/sql-pg, pg (PostgreSQL)
    • Authentication: jsonwebtoken (JWT)
    • Core: effect, @effect/platform-node

    Build Tools

    • turbo (Monorepo management), vite (Frontend), typescript (Type system)

  11. Use LoroDocMigration to evolve Loro document schemas

    main

    The LoroDocMigration schema transformer allows you to automatically upgrade older LoroDoc snapshots to the current schema version during decoding.

    When decoding a document, LoroDocMigration performs the following steps:

    1. Initializes a new LoroDoc.
    2. Imports the provided snapshot data.
    3. Checks the version field within the metadata map of the document.
    4. Iteratively applies all registered migrations from version 1 up to the current version.

    Note: Encoding via LoroDocMigration is forbidden to prevent accidental corruption of the migration logic; documents should be encoded using the standard AnyLoroDocSchema after migrations are complete.

    import { LoroDocMigration } from "@local/schema";

// When decoding data from a source (like a database or network),
// use LoroDocMigration to ensure the LoroDoc is upgraded to the latest version.
const result = LoroDocMigration.decode(snapshotData);

if (result._tag === "Success") {
  const upgradedDoc = result.value;
  // Use the upgraded doc...
} else {
  // Handle migration failure (e.g., missing version in metadata)
}

  12. Use the Sync service to manage data synchronization

    main

    The Sync service provides high-level methods to synchronize local CRDT data (using LoroDoc) with a remote server. It handles workspace management, authentication tokens, and local persistence via WorkspaceManager and Dexie.

    Available Methods

    • push({ workspaceId, snapshotId, snapshot }): Uploads a local snapshot to the server. If the workspace lacks a valid authentication token, the service automatically attempts to generate a new one via syncAuth.generateToken and updates the local workspace manager.
    • pull({ workspaceId }): Fetches the latest snapshot from the server for a specific workspace. This requires a valid token to be already present in the local workspace configuration.
    • join({ workspaceId }): A specialized method for new clients to join an existing workspace. It uses a clientId to request an initial state and an authentication token from the server, then initializes the local workspace with the returned snapshot and token.
    // Note: Sync is an Effect.Service, so it must be used within an Effect context

// Example: Joining a workspace
const joinResult = yield* Sync.join({
  workspaceId: "my-workspace-id"
});

// Example: Pushing a local snapshot
const pushResult = yield* Sync.push({
  workspaceId: "my-workspace-id",
  snapshotId: "snap-123",
  snapshot: myUint8ArraySnapshot
});

// Example: Pulling updates
const pullResult = yield* Sync.pull({
  workspaceId: "my-workspace-id"
});