DoltLite Documentation

The ext/misc directory contains several specialized extensions for DoltLite. These include virtual tables for interacting with external file formats and custom SQL functions for data processing:

• csv.c: A virtual table for reading Comma-Separated-Value (CSV) files.
• dbdump.c: A library providing functionality similar to the .dump command in the SQLite command-line shell.
• json1.c: SQL functions and table-valued functions for JSON processing (note: this is often already built into the SQLite amalgamation).
• rot13.c: A simple rot13() substitution function; useful as a template for creating custom SQL functions.
• series.c: An implementation of the generate_series virtual table; useful as a template for custom virtual table implementations.
• shathree.c: Implements sha3() and sha3_query() SQL functions.
• unionvtab.c: Implements unionvtab and swarmvtab virtual tables, allowing a single large table to be spread across multiple database files (with swarmvtab attaching files on demand).
• zipfile.c: A virtual table for reading and writing ZIP archives.

DoltLite uses various identifiers to reference specific states (snapshots) of the database. Commit refs are case-sensitive.

Common Revision Types

• Branches: main, feature/x (refers to the branch tip).
• Tags: v1.0.
• Commit Hashes: 40 lowercase hex characters (prefixes are not accepted).
• HEAD: The tip of the connection's current branch. Note that head (lowercase) is an error.
• Remote Tracking Branches: origin/main, remotes/origin/main, or refs/remotes/origin/main (available after dolt_clone or dolt_fetch).
• Ancestors:
  • <rev>~N: The Nth first-parent ancestor. ~ is shorthand for ~1.
  • <rev>^N: The Nth parent of a merge commit. ^ is shorthand for ^1. Note that ^0 is an error.
• Catalogs (Case-Insensitive):
  • WORKING: The working set (committed data plus staged and unstaged edits).
  • STAGED: The staged catalog, or HEAD if nothing is staged.

Lookup Order and Suffixes

When resolving a name, DoltLite follows this order: HEAD $\rightarrow$ hash $\rightarrow$ branch $\rightarrow$ tracking branch $\rightarrow$ tag. If a name is both a branch and a tag, the branch takes precedence. Suffixes like ~ and ^ chain from left to right (e.g., HEAD~2^2).

Catalog vs. Commit

WORKING and STAGED refer to a catalog, not a specific commit.

• Use them with: dolt_diff_*, dolt_diff_stat, dolt_diff_summary, dolt_schema_diff, dolt_patch, dolt_at_<table>, dolt_hashof_table, and dolt_hashof_db.
• Do NOT use them with: dolt_hashof, dolt_log, dolt_history_<table>, dolt_merge_base, or dolt_checkout (these require a specific commit).

# Examples of revision syntax
main              # Branch tip
v1.0               # Tag
40hexchars...      # Commit hash
HEAD               # Tip of current branch
WORKING            # The working set (case-insensitive)
STAGED             # The staged catalog (case-insensitive)

The Test Script Interpreter is designed to read and execute ASCII text files with a .test extension. These files contain SQL commands and expected results, which the interpreter uses to verify correctness and report discrepancies.

Key Behaviors:

• Isolation: Each script is evaluated independently. Context (like settings or state) does not carry forward between scripts.
• Cleanup: All open database connections are closed at the end of a script, and all database files created during the script's execution are deleted upon completion.
• Format: Scripts are processed line-by-line.

Compatibility and Parsing Rules:

To be compatible with this specification, a script must follow these rules:

• Module Names: Must not contain MODULE_NAME: or MIXED_MODULE_NAME:. If SCRIPT_MODULE_NAME: is present, the script is considered compatible.
• Properties: Must not contain REQUIRED_PROPERTIES: followed by non-whitespace text.
• Format Restrictions: Lines starting with | (indicating a dbtotxt format) are incompatible.
• No Preprocessor: Lines starting with # (C-preprocessor lines) are not supported; processing will be abandoned if they are encountered.
• Commands: Commands are identified by lines starting with exactly two minus signs followed by a lowercase letter (e.g., --command).
• Input Buffer: All lines that are not commands are accumulated into an "input buffer" which commands can access or reset.

DoltLite supports multiple connections and processes on a single database file, but it is not a multi-writer server. Coordination is managed via a graph lock sidecar that serializes durable writers.

Key concurrency behaviors include:

• One durable writer at a time: A connection holding an explicit write transaction owns the graph lock. Other processes attempting to write will receive SQLITE_BUSY (or a retryable busy class). This applies to both standard SQL (e.g., INSERT) and version-control commands (e.g., dolt_branch, dolt_tag, dolt_commit).
• Read-only connections never block writers: Reads (including dolt_status) do not take an exclusive lock. A second process polling status will not cause a writer's INSERT or dolt_commit to fail.
• Snapshot-safe write upgrades: If a transaction has established a read snapshot, it cannot upgrade to a writer if a peer has advanced the store in the meantime. This upgrade attempt will return SQLITE_BUSY_SNAPSHOT to prevent mixing catalogs.
• GC and VACUUM cooperation: dolt_gc or VACUUM may be deferred or report busy while a writer holds the graph lock. Once the writer finishes, GC completes safely.
• Conflicts are transient: Conflicted merges exist only within the transaction that produced them. A dolt_commit will be refused if conflicts remain, and no conflicted state is persisted to disk for other connections to inherit.