Reverse Linear Sync Engine Study

LSE synchronizes client-side changes to the server using a transaction-based system. When a model property is changed, the in-memory model is updated immediately, but the local database (IndexedDB) is not modified until the server confirms the change via a delta packet.

The Synchronization Lifecycle:

1. Property Assignment: Setting a property triggers observability (via M1), recording the property name, old value, and new value.
2. Transaction Creation: Calling model.save() creates an UpdateTransaction capturing these changes.
3. Queueing & Caching: The transaction is added to a request queue and cached in the __transactions table in IndexedDB for persistence.
4. Batching: The TransactionQueue batches multiple transactions (sharing the same batchIndex) into a single GraphQL mutation to reduce network requests.
5. Execution: The TransactionExecutor sends the merged GraphQL mutation to the server.
6. Completion: Upon a successful server response (containing lastSyncId), the transaction is cleared from the cache. The local database is only updated once the corresponding delta packet arrives from the server.

issue.assignee = user;
issue.save();

The TransactionQueue manages the lifecycle of transactions using four distinct internal arrays:

1. createdTransactions: Newly created transactions. A microtask scheduler (commitCreatedTransactions) moves these to queuedTransactions and increments the batchIndex. Transactions created in the same event loop share a batchIndex and are eligible for batching.
2. queuedTransactions: Transactions waiting to be executed. They are persisted in the __transactions IndexedDB table. The scheduler moves them to executingTransactions in batches based on batchIndex, independence, and size limits.
3. executingTransactions: Transactions that have been sent to the server but are awaiting a response.
4. persistedTransactionsEnqueue: Transactions loaded from the __transactions table during database bootstrap. After remote updates are processed, these are moved to queuedTransactions.

Note: There is also a completedButUnsyncedTransactions queue used for transactions that have been accepted by the server but are waiting for a specific lastSyncId to arrive via a delta packet.

Undo and redo in the Linear Sync Engine (LSE) are transaction-based. Every transaction type implements an undoTransaction method. This method performs the reversal logic and returns a new transaction that can be used for the redo operation.

Key behaviors:

• Transaction Reversal: For example, an UpdateTransaction reverts a model property to its previous value and returns a new UpdateTransaction to the UndoQueue to enable redo.
• Synchronization: When undoTransaction executes, it creates a new transaction and adds it to queuedTransactions to ensure the state remains synchronized.
• UndoQueue Logic: The UndoQueue determines which operations to track by subscribing to the transactionQueuedSignal. When an edit is made via UndoQueue.addOperation, the queue listens for the next signal emitted when transactions are added to queuedTransactions. Once the operation's callback finishes and save() is called, the resulting transactions are pushed to the undo/redo stack.
• Redo Support: Performing an undo triggers the undoTransaction method, which provides the necessary transaction for the redo stack.

// Example of how the UI triggers an undoable operation
n.title !== d &&
  o.undoQueue.addOperation(
    s.jsxs(s.Fragment, {
      children: [
        "update title of issue ",
        s.jsx(Le, {
          model: n,
        }),
      ],
    }),
    () => {
      (n.title = d),
        n.save();
    }
  );

After raw models are written to the ObjectStore, LSE performs model hydration to make them accessible in memory.

1. Instant Hydration: LSE calls Database.getAllInitialHydratedModelData to load models with a loadStrategy of instant.
2. Instantiation: For each model, LSE retrieves the constructor from the ModelRegistry and instantiates the object.
3. Population: Instead of passing data to the constructor, LSE initializes the object and then calls updateFromData to populate it. It also calls attachToReferencedProperties to resolve references.
4. Object Pool: Hydrated objects are added to the Object Pool via addModelToLiveCollections. The Object Pool is implemented as a modelLookup map on SyncClient, which maps model IDs to their in-memory objects for efficient retrieval.

Lazy Hydration: To save memory, LSE uses lazy hydration for non-essential data. Data is fetched via network or local queries only when needed (e.g., loading comments when viewing an issue). Classes that support this implement a hydrate method, including Model, LazyReferenceCollection, LazyReference, RequestCollection, and LazyBackReference.

When a full bootstrap is initiated, LSE uses GraphQLClient.restModelsJsonStreamGen to request data from the server.

Request Format: The request is a GET request to the /sync/bootstrap endpoint with two query parameters:

• type: Set to "full".
• onlyModels: A comma-separated list of model names (derived from modelsToLoad).

Response Format: The response is a stream of JSON objects. Each line (except the last) represents a single model instance. The final line contains a special metadata object prefixed with _metadata_=.

Metadata Fields:

• method: The source of the data (e.g., "mongo").
• lastSyncId: The snapshot ID the client is now synchronized to.
• subscribedSyncGroups: The sync groups the client should subscribe to for incremental changes.
• databaseVersion: The version of the database schema.
• returnedModelsCount: A map of model names to the count of instances returned, used to verify request validity.

// Example of a model instance in the stream
{
  "id": "556c8983-ca05-41a8-baa6-60b6e5d771c8",
  "createdAt": "2024-01-22T01:02:41.099Z",
  "updatedAt": "2024-05-16T08:23:31.724Z",
  "number": 1,
  "title": "Welcome to Linear 👋",
  "priority": 1,
  "boardOrder": 0,
  "sortOrder": -84.71,
  "startedAt": "2024-05-16T08:16:57.239Z",
  "labelIds": ["30889eaf-fac5-4d4d-8085-a4c3bd80e588"],
  "teamId": "89388c30-9823-4b14-8140-4e0650fbb9eb",
  "projectId": "3e7ada3c-f833-4b9c-b325-6db37285fa11",
  "projectMilestoneId": "397b95c4-3ee2-47b0-bad1-d6b1c7003616",
  "subscriberIds": ["4e8622c7-0a24-412d-bf38-156e073ab384"],
  "previousIdentifiers": [],
  "assigneeId": "4e8622c7-0a24-412d-bf38-156e073ab384",
  "stateId": "030a7891-2ba5-4f5b-9597-b750950cd866",
  "reactionData": [],
  "__class": "Issue"
}

// Example of the metadata line
_metadata_={"method":"mongo","lastSyncId":2326713666,"subscribedSyncGroups":["89388c30-9823-4b14-8140-4e0650fbb9eb"],"databaseVersion":948,"returnedModelsCount":{"Issue":3}}