sqllogictest-rs

repository·main·Indexed 42 days ago

https://github.com/risinglightdb/sqllogictest-rs

A Rust testing framework for verifying SQL database correctness. It provides a parser and runner library, as well as the `sqllogictest-bin` CLI tool for executing `.slt` test files against engines like Postgres and MySQL. Features include parallel execution with database isolation, JUnit reports, variable substitution, and support for custom database drivers via a JSON-based protocol using `ExternalDriver`.

Tokens
10.7K
Snippets
37
Records
53
Agent score
53%

What's inside sqllogictest-rs

  1. Overview of Sqllogictest

    main
    Sqllogictest is a testing framework designed to verify the correctness of an SQL database. This Rust implementation provides a parser and runner library for the sqllogictest format, allowing developers to run SQL test scripts against their database engines to ensure compliance with expected behaviors.
  2. Retry failed queries or statements

    main

    You can add a retry clause to handle transient failures. The syntax is retry <count> backoff <duration>.

    Note: The retry clause cannot be used with single-line regex error message extensions due to syntax limitations.

    Example

    query I retry 3 backoff 5s
    SELECT id FROM test;
    ----
    1
  3. Use <slt:ignore> to skip volatile output parts

    main

    When testing statements like EXPLAIN, parts of the output (like costs or row counts) might change frequently. Use the <slt:ignore> tag to skip these volatile segments.

    query T
    EXPLAIN SELECT * FROM foo;
    ----
    Seq Scan on t  (cost=<slt:ignore> rows=<slt:ignore> width=<slt:ignore>)
       Filter: (x > 1)  
  4. How to test for expected errors in .slt files

    main

    You can verify that a statement or query fails with specific error criteria using the error keyword.

    Error Matching Modes

    • No message check: [statement|query] error (only checks that an error occurred).
    • Regex match: [statement|query] error <regex> (checks if the error message matches a regular expression).
    • SQLSTATE match: [statement|query] error (<SQLSTATE>) (checks for an exact SQLSTATE match).
    • Exact multiline match: Use ---- to provide the expected error message. The message must end with two consecutive empty lines. Leading/trailing whitespace is ignored, and non-consecutive empty lines are allowed.

    Examples

    # Regex match
    statement error Multiple object drop not supported
    DROP VIEW foo, bar;
    
    # SQLSTATE match
    statement error (42P01)
    SELECT * FROM non_existent_table;
    
    # Exact multiline match
    query error
    SELECT 1/0;
    ----
    db error: ERROR: Failed to execute query
    
    Caused by these errors:
    1: Failed to evaluate expression: 1/0
    2: Division by zero
    
    
    # Ensure that the statement errors and that the error
    # message contains 'Multiple object drop not supported'
    statement error Multiple object drop not supported
    DROP VIEW foo, bar;
    
    # Ensure that the statement errors and that the SQLSTATE is exactly the expected one.
    statement error (42P01)
    SELECT * FROM non_existent_table;
    
    # The output error message must be the exact match of the expected one to pass the test...
    query error
    SELECT 1/0;
    ----
    db error: ERROR: Failed to execute query
    
    Caused by these errors:
    1: Failed to evaluate expression: 1/0
    2: Division by zero
    
    
  5. Run external shell commands with 'system'

    main

    The system record allows running shell commands to manipulate external resources during a test.

    • system ok: Checks if the command exits with code 0.
    • Output checking: To check the command's output, provide the expected output after ----. Like error messages, the output must end with two consecutive empty lines.
    • Environment Variables: Standard shell environment variables (e.g., $USER) are supported.

    Examples

    # Fails if exit code is not 0
    system ok
    exit 1
    
    # Check command output
    system ok
    echo $'Hello\n\nWorld'
    ----
    Hello
    
    World
    
    
    
    # Use environment variables
    system ok
    echo $USER
    ----
    xxchan
    system ok
    exit 1
    
    system ok
    echo $'Hello\n\nWorld'
    ----
    Hello
    
    World
    
    
    
    system ok
    echo $USER
    ----
    xxchan
  6. Enable variable substitution and 'let' bindings

    main

    To use variables in your tests, you must enable substitution by adding control substitution on to the top of your .slt file.

    Variable Substitution

    Supports standard shell-like syntax:

    • $foo (short)
    • ${foo} (long)
    • ${bar:default} (with default value)

    Special Variables:

    • $__TEST_DIR__: Path to a temporary directory specific to the current test case.
    • $__NOW__: Current Unix timestamp in nanoseconds.
    • $__DATABASE__: Name of the current database.

    Binding results with 'let'

    Use the let keyword to capture results from a query into variables for use in subsequent statements. Requirements:

    • The query must return exactly 1 row.
    • The number of columns must match the number of variables.
    • You must have control substitution on enabled to use the variables later.

    Example

    control substitution on
    
    let id
    SELECT id FROM users WHERE name = 'alice'
    
    query TTT
    SELECT $id
    ----
    1
    control substitution on
    
    # Execute a query that returns exactly 1 row with 1 column, bind result to 'id'
    let id
    SELECT id FROM users WHERE name = 'alice'
    
    # Multiple variables can be bound from a single query
    let user_id, user_name, user_email
    SELECT id, name, email FROM users WHERE id = 1
    
    # Use the bound variable in subsequent queries
    query TTT
    SELECT $user_id, $user_name, $user_email
    ----
    1 alice alice@example.com
  7. Install and use the sqllogictest-bin CLI

    main

    The sqllogictest-bin crate provides a CLI tool for running .slt test files against supported databases (currently only postgres and mysql). It supports parallel execution with database isolation, colorful diffs, JUnit reports, and automatic test file updates.

    To install the binary:

    cargo install sqllogictest-bin

    Common usage patterns:

    • Run all .slt files in a directory against a default Postgres connection: sqllogictest './test/**/*.slt'
    • Run tests and automatically update the expected output in the .slt files to match the actual database output: sqllogictest './test/**/*.slt' --override
    # run scripts in `test` directory against postgres with default connection settings
    sqllogictest './test/**/*.slt'
    
    # run the tests, and update the test files with the actual output!
    sqllogictest './test/**/*.slt' --override
  8. How to write .slt test files

    main

    .slt files use a dialect of Sqlite to verify SQL database correctness. Tests consist of statements or queries followed by expected results, separated by ----.

    Basic Syntax

    • Comments: Start with #.
    • Successful Statements: Use statement ok followed by the SQL.
    • Successful Queries: Use query <type> [options] followed by the SQL and ---- then the expected output.
      • <type>: e.g., II (two integer columns), T (text/string columns).
      • rowsort: An option to sort output before comparison.

    Example: Successful Query

    # 'II' means two integer output columns
    # rowsort means to sort the output before comparing
    query II rowsort
    SELECT * FROM foo;
    ----
    3 4
    4 5
  9. Understanding statement output expectations

    main

    In .slt files, you can control how statement execution results are recorded:

    • statement ok: Indicates you do not care about the number of affected rows. The runner will not update the count in the test file even if the actual count differs.
    • statement count <n>: Expects exactly <n> rows to be affected. If the actual count differs from <n>, the runner will update the test file with the real count.
    • statement error <pattern>: Expects the statement to fail. The runner will check if the error message matches the provided pattern and may update the file with the full error message if it doesn't match.
  10. How retry logic works in sqllogictest

    main

    The run_async method supports automatic retries for specific records. If a Record::Statement, Record::Query, or Record::System is configured with a retry policy, the runner will attempt to execute the record multiple times before failing.

    • Attempts: The number of times to try the operation.
    • Backoff: The duration to wait between attempts.

    If the record succeeds on any attempt, the runner returns the successful result. If all attempts fail, the runner returns the error from the final attempt.

  11. Configure statement retries with retry and backoff

    main

    You can specify an optional retry configuration for statements or queries using the retry keyword followed by the number of attempts and a backoff duration. The duration should be a human-readable string (e.g., 1s, 500ms).

    Example syntax:

    SELECT * FROM table retry 3 backoff 1s
    SELECT * FROM foo retry 3 backoff 1s