Replicate Sync Engine

The architecture uses a dual-storage pattern to balance standard Convex CRUD with offline synchronization capabilities.

1. Main Table: This is your primary application data. Online clients interact with this table directly using standard Convex queries, mutations, and subscriptions. It is the single source of truth for the online experience.
2. Deltas Table: This acts as an append-only operation log. It records every insert, update, and delete as a Yjs delta (or a synthetic marker). This table is not used during normal online operation; it is exclusively used by clients to recover and synchronize state after being offline.

When a client is online, it follows the standard Convex flow. When a client reconnects after being offline, it compares its local state to the Main Table, identifies out-of-sync documents, and requests missing deltas from the Deltas Table to resolve conflicts via Yjs.

When performing optimistic updates (applying local changes before server confirmation), do not use manual 'rollback' logic (e.g., calling a clearDeleted function in a .catch block). Manual rollbacks are prone to race conditions where a subscription update might be skipped during the mutation, leaving the client in a stale state even after the 'rollback' completes.

Best Practice: Use a transactional boundary where changes are staged and only applied upon a successful commit. If the server call fails, the entire transaction should be automatically rolled back to the previous known good state, similar to the TanStack DB pattern.

// Recommended pattern: Staged changes with automatic rollback
const tx = createTransaction({
  mutationFn: async () => {
    await fetch('/api/delete', { ... });
  }
});

tx.mutate(() => {
  collection.delete("doc-1"); // Staged, not applied yet
});

await tx.commit(); // Only applied if mutationFn succeeds; otherwise, rolled back

The synchronization system relies on three primary Convex endpoints:

1. delta (Query):

   • Streaming mode ({ seq, limit? }): Reactively returns CRDT changes since a specific sequence number.
   • Recovery mode ({ document, vector }): Used on startup to fetch only missing deltas based on a local Yjs state vector.

2. replicate (Mutation): Handles all write operations. The client sends one of the following shapes:

   • { type: 'insert', document: id, bytes: crdtUpdate, material: { ...fields } }
   • { type: 'update', document: id, bytes: crdtUpdate, material: { ...fields } }
   • { type: 'delete', document: id, bytes: crdtUpdate } The server runs authorization, updates the Convex table, stores the delta, and triggers side-effect hooks.

3. presence (Mutation): Manages session state via four actions:

   • join: Register a session.
   • leave: Deregister a session.
   • mark: Report sync progress (current seq/vector).
   • signal: Heartbeat and cursor/profile updates.

The Read Path describes how the client receives updates from the server via WebSockets.

1. Subscription: The client subscribes via WebSocket using onUpdate(stream, { cursor }).
2. Query: The server queries for new changes: SELECT * FROM documents WHERE seq > cursor.
3. Application: Upon receiving changes, collection.ts applies the updates using Y.applyUpdate.
4. Persistence: The updates are upserted/inserted/deleted into the local TanStack DB.
5. Notification: The actorManager.onServerUpdate(documentId) is called, and the Actor receives an ExternalUpdate to update its stored vector for bookkeeping.

/* Conceptual flow of the Read Path */
// 1. Subscribe: onUpdate(stream, { cursor })
// 2. Server Query: SELECT * FROM documents WHERE seq > cursor
// 3. Receive changes
// 4. collection.ts applies Y.applyUpdate
// 5. ops.upsert/insert/delete to TanStack DB
// 6. actorManager.onServerUpdate(documentId)
// 7. Actor receives ExternalUpdate
// 8. Update stored vector

The Replicate sync system uses a client-server architecture designed for real-time collaborative editing using Yjs.

Client Side:

• Maintains a Y.Doc.
• Tracks a clientID and State Vector.
• Uses SQLite for local persistence.
• Manages Effect.ts and Actors.
• Communicates via WebSocket (WS) sync and heartbeats.

Server Side (Convex):

• Manages three primary data collections: documents (deltas), snapshots, and sessions.
• Runs COMPACTION LOGIC to merge deltas into snapshots.
• Orchestrates synchronization between clients and the persistent state.

┌─────────────────────────────────────────────────────────────────────────┐
│                         SYSTEM ARCHITECTURE                              │
├─────────────────────────────────────────────────────────────────────────┤
│                                                                          │
│  ┌─────────────┐         ┌─────────────────────────────────────────┐    │
│  │   CLIENT    │         │              SERVER (Convex)             │    │
│  ├─────────────┤         ├─────────────────────────────────────────┤    │
│  │             │         │                                         │    │
│  │  Y.Doc      │◄───────►│  ┌─────────┐  ┌──────────┐  ┌────────┐ │    │
│  │  clientID   │   sync   │  │documents│  │ snapshots│  │sessions│ │    │
│  │             │   WS     │  │ (deltas)│  │          │  │        │ │    │
│  │  State      │         │  └────┬────┘  └────┬─────┘  └───┬────┘ │    │
│  │ Vector      │────────►│       │            │            │      │    │
│  │             │ heartbeat │       │            │            │      │    │
│  │  SQLite     │         │       └────────────┼────────────┘      │    │
│  │ (persist)   │         │                    │                   │    │
│  │             │         │              ┌─────┴─────┐             │    │
│  │  Effect.ts  │         │              │COMPACTION │             │    │
│  │  Actors     │         │              │  LOGIC    │             │    │
│  └─────────────┘         │              └───────────┘             │    │
│                           └─────────────────────────────────────────┘    │
│                                                                          │
└─────────────────────────────────────────────────────────────────────────┘