DOCSEARCHbyAlgolia
Submit Docs

Dolt SQL Database

repository·main·Indexed 12 days ago

https://github.com/dolthub/dolt

Dolt is a SQL database that incorporates Git-like version control, allowing users to fork, clone, branch, merge, and push data repositories. It is compatible with MySQL clients and provides a specialized CLI for managing data as versioned tables, including core version control commands like commit, log, and merge, as well as SQL and server management tools.

Tokens
66K
Snippets
251
Records
354
Agent score
98%

What's inside Dolt

  1. Use the go-sql-server-driver for MySQL port integration tests

    main

    The go-sql-server-driver package is a driver and test suite designed for testing interactions with dolt sql-server. It is intended for declarative and robust integration tests that target the exposed MySQL port of the sql-server.

    Use this package when your tests primarily involve executing SQL queries or interacting with the database engine via a MySQL protocol. If your tests instead focus on the dolt binary itself (such as CLI command validation, exit codes, or help text), you should use bats instead.

  2. Catalog of corrupt databases for testing `fsck`

    main
    This directory contains a collection of intentionally corrupted Dolt databases designed to test the behavior of the fsck tool and other future diagnostic utilities. Each database simulates a specific type of data corruption or structural failure, such as mismatched commit contents, bad CRC values in journals, or missing schema/table objects.

  3. What is Noms Block Store (NBS)?

    main

    NBS is a horizontally-scalable storage backend optimized for the Noms database. It provides storage for a content-addressed Directed Acyclic Graph (DAG) of nodes. Each node is addressed by a 20-byte hash of its byte-sequence.

    Key characteristics include:

    • Immutability: There are no update or delete operations; instead, the system uses insert, update root, and garbage collect.
    • Durability: Inserting a novel byte-sequence is only considered durable once the root is updated.
    • Concurrency: Supports file-level multiprocess concurrency with optimistic locking for multiple writers.
    • Deduplication: NBS efficiently detects and drops duplicate chunks, so writers do not need to manage duplicate data manually.

  4. What is Dolt?

    main
    Dolt is a SQL database that provides Git-like version control capabilities for data. While Git versions files, Dolt versions tables. It allows you to fork, clone, branch, merge, push, and pull databases just like a Git repository. You can interact with it using a Git-like CLI or by connecting via any MySQL-compatible client.

  5. What is Dolt and how does it work?

    main

    Dolt is a SQL database that implements Git-like version control for data. While Git versions files, Dolt versions tables.

    Key capabilities include:

    • Version Control: You can fork, clone, branch, merge, push, and pull databases just like Git repositories.
    • SQL Interface: Connect to Dolt like any MySQL database to run queries or update data using standard SQL commands.
    • CLI Operations: Use the Dolt CLI to import CSV files, commit changes, manage branches, and resolve merge conflicts.

    Dolt is compatible with MySQL clients and can be used with remote hosting services like DoltHub (public data hosting) or DoltLab (self-hosted version).

  6. Understand the archive test repository variants

    main

    The archive-test-repos directory contains several repository variants used to test different storage and archive scenarios:

    • base: The initial repository containing only archive files.
    • large_clone: A full clone of base with significant mutations and subsequent garbage collection. It contains over 1000 new chunks, used to test the conversion of Snappy objects to ZStd compressed objects during push when archives are enabled.
    • small_clone: A full clone of base with minimal mutations. It is used to test scenarios where archive files contain Snappy objects.
    • v1: A copy of base used to verify compatibility with version 1 of the archive format.
    • v2: A repository used to verify compatibility with version 2 of the archive format.

  7. How Dolt version control works via SQL

    main

    Dolt provides version control functionality through two primary interfaces: a Git-style CLI and a SQL interface.

    In the SQL interface, version control operations are exposed as:

    1. Stored Procedures: Used for write operations (e.g., dolt_add, dolt_commit). These follow the dolt_<command> naming pattern.
    2. System Tables: Used for read operations (e.g., dolt_log). These allow you to inspect the history and lineage of your data.

    When using stored procedures, options are passed similarly to the CLI. For named arguments like a commit message, use two arguments in sequence, such as ('-m', 'your message').

    Note: A Dolt commit is distinct from a standard SQL COMMIT. To automatically generate a Dolt commit for every transaction, configure the system variable @@dolt_transaction_commit.

    -- Example of adding tables and committing via SQL
CALL dolt_add('table1', 'table2');
CALL dolt_commit('-m', 'My commit message');

-- Example of viewing history
SELECT * FROM dolt_log;

  8. Define import benchmark inputs in YAML

    main

    Benchmark tests are defined in YAML files specifying repositories (servers) and table specifications.

    Repository Types

    • Dolt server: Specify using the server field (e.g., with a port).
    • Dolt CLI: Omit the server field.
    • MySQL server: Use the external-server field.

    Table Specification Options

    • fmt (string): The file format for importing. Options: csv, sql.
    • rows (int): Number of rows to import.
    • schema (string): The CREATE_TABLE statement.
    • shuffle (bool): If true, generated rows are shuffled (default is sorted).
    • batch (bool): Whether to batch insert statements (only applies when fmt: sql).

    Server Lifecycle

    • Dolt sql-server: A new instance is constructed individually for each test run.
    • External servers: Managed outside the benchmarker; the same instance is reused for every table import test.
    • Caching: Import files are cached based on schema, rows, and fmt between tests.

  9. Configure NBS storage backends (Local vs. AWS)

    main

    NBS can be configured using two primary backends depending on your requirements for scale and availability:

    1. Local Disk Backend: Optimized for performance and multiprocess concurrency. It is significantly faster than LevelDB for Noms workloads because it locates related chunks together rather than maintaining key-order.
    2. AWS Backend: Stores data primarily in Amazon S3 with a single DynamoDB item. This configuration provides high availability (comparable to S3/DynamoDB) and a cost profile similar to S3 while maintaining Noms' consistency guarantees.

  10. Examine changes in the working set

    main

    Before committing changes, you can inspect your current working set using the dolt_status system table and table-specific diffs using dolt_diff_<tablename> functions.

    • dolt_status: Shows which tables have been modified and whether those changes are staged.
    • dolt_diff_<tablename>: Provides a row-level diff showing the transition from the last commit to the current working state.
    -- Check status of all tables
SELECT * FROM dolt_status;

-- View row-level diff for a specific table
SELECT * FROM dolt_diff_employees;

  11. Understand the Data Dump Loading Test implementation

    main

    The Data Dump Loading tests verify Dolt's ability to import data dumps generated by mysqldump.

    Key details:

    • Test Data: The tests currently use a modified version of the MySQL Sakila Database dump file (sakila_dump.sql).
    • Modifications: Certain unsupported syntax parts in the original Sakila dump have been commented out (marked with UNSUPPORTED SYNTAX).
    • Scope: The dump file excludes original stored functions and procedures, though one specific procedure is included to test Dolt's procedure functionality.
    • Output: Successful test runs will output status messages for specific scenarios, such as import mysqldump: empty database dump or import mysqldump: a simple table dump.

  12. What are Workbench stability tests?

    main

    The tests located in workbenchTests are designed to ensure the stability of the SQL workbench used on Hosted.

    Because the workbench relies heavily on Dolt system tables, functions, and procedures, these tests act as a safeguard against breaking changes to those interfaces. Changes to Dolt's internal SQL interfaces that impact these queries are monitored to ensure the workbench remains functional.