Grist Core

The grist-core repository provides a powerful server for hosting spreadsheets. Key features include:

Data Model

• Hybrid Database/Spreadsheet: Columns are named and typed (like a database), but can be filled with spreadsheet-style formulas that update automatically.
• Portable SQLite Format: Documents are based on SQLite, making them easy to back up, restore, and move between different hosts.

Formulas and Logic

• Python Formulas: Supports full Python syntax and the standard library, alongside many Excel functions.
• AI Formula Assistant: Supports formula generation using OpenAI, Llama (via OpenRouter), or other OpenAI-compatible endpoints.
• Formula Timer: A tool for diagnosing slow-performing formulas.

UI and Visualization

• Dashboards: Drag-and-drop layout using widgets like Charts, Card views, Calendar widgets, and Summary tables.
• Widget Linking: Allows widgets to filter and edit data in sync.
• Formatting: Includes Conditional Formatting, Markdown in text cells, and specialized editors for Dates, Toggles, and Currency.

Integrations and Automation

• REST API: Includes an interactive API console and support for service accounts.
• Webhooks: Configurable outgoing webhooks with support for authorization headers and formula-based triggers.
• Data Import/Export: Supports CSV, Excel, Google Drive, and direct imports from Airtable. Supports incremental imports to avoid duplication.
• Native Forms: Create forms that feed directly into spreadsheets, supporting file attachments and URL parameter pre-population.

Collaboration and Security

• Access Control: Control access at the document, workspace, or even individual row, column, and table level.
• Collaboration: Real-time viewing, cell comments with @-mentions, and a 'Suggest changes' workflow.
• Sandboxing: Supports gVisor (Linux/Docker), native sandboxing (macOS), or Wasm-based sandboxing (Deno/Pyodide) for untrusted documents.

Grist distinguishes between high-level user intentions and low-level data mutations.

User Actions

Created in the frontend (e.g., a user typing in a cell), these are sent via WebSocket to the Node.js server. They represent intent (e.g., UpdateRecord, AddRecord, RenameTable). The authoritative list of these is defined in sandbox/grist/useractions.py using the @useraction decorator.

Doc Actions

When the Python data engine processes a User Action, it converts it into one or more Doc Actions. Doc Actions are strict, simple data or schema changes (e.g., the result of a formula calculation).

The Workflow

1. Client sends User Action via WebSocket.
2. Node.js forwards it to the Python Data Engine.
3. Python Data Engine produces Doc Actions.
4. Node.js updates the local SQLite file using the Doc Actions and syncs to S3.
5. Node.js broadcasts the Doc Actions to all connected browsers via WebSocket.
6. Clients update their local in-memory state using the Doc Actions.

Grist can be deployed as a single server or as a distributed system composed of specialized components. A scalable deployment uses the following architecture:

Core Components

• Home Servers: Handle user requests related to document management (listing documents, access control, sharing, and API requests). They communicate with HomeDB, Redis, and Doc Workers but do not open documents themselves.
• Doc Workers: Responsible for in-document interactions. Each open document is assigned to exactly one Doc Worker. The worker maintains a local SQLite file and a sandboxed Python interpreter (the "data engine") to evaluate formulas.
• Application Load Balancer (ALB): Manages SSL and routes HTTP requests to Home Servers and WebSocket connections to Doc Workers.

Storage and State

• Home DB (Postgres): Stores metadata such as users, organizations, workspaces, documents, sharing permissions, and billing.
• S3 (or S3-compatible stores like MinIO): Used for long-term storage of .grist (SQLite) files. Doc Workers fetch files from S3 when a document is opened and sync them back periodically.
• Redis: Tracks available Doc Workers and manages the mapping of open documents to specific workers.

Grist contains legacy code using Knockout.js observables and modern code using GrainJS observables. The primary difference is how dependencies are tracked in computed values.

Knockout.js (Legacy)

Dependencies are created implicitly when the observable is called within a computed function.

GrainJS (Modern)

Dependencies are created explicitly by using the use() callback provided to the computed function. This makes the data flow more predictable.

// Knockout (Legacy)
import * as ko from 'knockout';
const kObs = ko.observable(17);
const kComputed = ko.computed(() => kObs() * 10); // Implicit dependency

// GrainJS (Modern)
import {Computed, Observable} from 'grainjs';
const gObs = Observable.create(null, 17);
const gComputed = Computed.create(null, use => use(gObs) * 10); // Explicit dependency via use()

When a user opens a document, the following lifecycle occurs:

1. Assignment: The Home Server assigns the document to a specific Doc Worker.
2. Fetching: The Doc Worker fetches the .grist (SQLite) file from S3 to its local disk.
3. Data Engine Initialization: The Doc Worker starts a sandboxed Python process (the data engine). All document data (except for tables marked as "on-demand") is loaded from the SQLite file into the Python data engine's memory.
4. Client Connection: The browser connects to the Doc Worker via WebSockets. The browser fetches metadata tables (prefixed with _grist_) and other necessary table data, maintaining an in-memory representation of the document that stays in sync via WebSocket updates.

Note on Memory: Data is typically held in memory in three places: the Python data engine, the Node.js Doc Worker, and the browser's JavaScript environment.

The Grist front-end state is organized into a hierarchical model called DocModel. This model drives the entire UI.

Model Structure:

• DocModel: The root container for the document.
• MetaTableModel: Contains metadata tables used by the Grist application itself.
  • MetaRowModel: Represents a row in a metadata table. These are enhanced with specific computed fields based on their table type.
• DataTableModel: Represents user-data tables (treated generically).
  • DataRowModel: Represents a row in a user table.
• BaseRowModel: The base class for both MetaRowModel and DataRowModel.

Key Implementation Detail:

Every RowModel contains an observable for each field. While these are legacy Knockout observables, they are compatible with modern GrainJS code; you can use a Knockout observable as a dependency in a GrainJS Computed or pass it to GrainJS dom methods.