DOCSEARCHbyAlgolia
Submit Docs

toasty

repository·main·Indexed 25 days ago

https://github.com/tokio-rs/toasty

An async ORM for Rust (v0.9.0) supporting SQL (SQLite, Turso, PostgreSQL, MySQL) and NoSQL (DynamoDB) databases. Toasty focuses on preserving native database behaviors through pass-through and backend-specific methods rather than providing a generic abstraction. It includes a query engine, procedural macros for model definition, and a CLI for managing database migrations.

Tokens
116.1K
Snippets
316
Records
547
Agent score
78%

What's inside toasty

  1. Overview of Toasty example capabilities

    main

    Toasty provides several specialized examples to demonstrate different feature sets:

    • quickstart-blog: Defining models/keys and CRUD operations over has_many/belongs_to relationships.
    • forum-relationships: has_one relations, bidirectional traversal, preloading with .include(), multi-step via relations, and association filters.
    • product-search: Data reading including filter expressions, sorting, limit/offset, cursor pagination, column projection, and composite indexes.
    • cms-article-fields: Field options like create/update defaults, auto timestamps, Json<T>, queryable Vec<scalar>, deferred columns, and custom table/column names.
    • crm-embedded: Embedded value types (flattened structs, tagged-union enums, newtype keys) and partial updates with stmt::patch.
    • store-operations: Writes including interactive transactions, savepoints, batch inserts, query-based updates/deletes, and raw SQL.
    • service-ops: Project layout involving shared models in a library, an application binary, and a migration CLI.

  2. Overview of Toasty crates and responsibilities

    main

    Toasty is organized into several specialized crates within a Cargo workspace:

    • toasty: The primary user-facing crate containing the query engine, runtime, typed statement builders (stmt/), relationship abstractions (relation/), and the Model trait.
    • toasty-core: Contains shared types used by all crates, including schema representations (app, db, mapping), the Statement AST (stmt/), and the driver interface.
    • toasty-macros: Provides procedural macros like #[derive(Model)] and #[derive(Embed)] to generate model implementations, query builders, and field accessors.
    • toasty-driver-*: Specific implementations for different databases (SQLite, PostgreSQL, MySQL, DynamoDB).
    • toasty-sql: A utility crate used by SQL-based drivers to convert the Statement AST into dialect-specific SQL strings.

  3. Understand the Toasty Query Execution Model

    main

    The Toasty engine executes ORM operations by transforming a Statement AST into a sequence of executable actions (a 'mini program') interpreted by a runtime.

    Key components of the execution model:

    • Instructions (Actions): Discrete operations such as ExecSQL, Filter, or NestedMerge.
    • Variables: Storage slots (registers) used to hold intermediate results between instructions.
    • Linear Execution: Instructions run in sequence. While currently linear, the interpreter is designed to eventually execute independent operations in parallel.
    • Interpreter: The executor that reads instructions, fetches inputs from variables, performs the operation, and stores outputs back to variables.
    // Example of a compiled program for loading users with todos:
$0 = ExecSQL("SELECT * FROM users WHERE ...")
$1 = ExecSQL("SELECT * FROM todos WHERE user_id IN ...")
$2 = NestedMerge($0, $1, by: user_id)
return $2

  4. Relationship types in Toasty

    main

    Toasty supports several relationship patterns for modeling data connections:

    • BelongsTo: Defines foreign keys, allows accessing the parent, and setting the relation on creation.
    • HasMany: Allows querying children, creating through the relation, inserting/removing, and performing scoped queries.
    • Many-to-Many: Uses a join model to allow traversing in both directions, filtering by endpoints or join metadata, and changing links.
    • HasOne: Handles required vs optional relations, creating/updating the child, and replace/unset behavior.
    • Preloading Associations: Uses the .include() method to load relations upfront and avoid extra queries (N+1 problem).

  5. Understand the Query Engine Execution Flow

    main

    When you call db.exec(statement), the engine processes the statement through several stages:

    1. Verification: (In debug mode) Validates the statement structure.
    2. Lowering: Transforms the statement from a high-level model-based representation (e.g., SELECT MODEL FROM User) to a low-level table-based representation (e.g., SELECT id, name, email FROM users).
    3. Planning: Translates the lowered representation into a series of driver operations and assigns variables with specific types (Type::List or Type::Unit).
    4. Execution: Runs the plan using a VarStore that holds type information to ensure the resulting value stream conforms to the expected types.

  6. Understand Document and Collection Field Storage

    main

    Toasty uses the #[document] attribute to store #[derive(Embed)] structs as single document columns. This allows for nested data structures within a single database field.

    Storage Mapping by Backend:

    • PostgreSQL: jsonb
    • MySQL: JSON
    • SQLite: JSON text
    • DynamoDB: A Map M attribute

    Collections:

    • Vec collections of embeds are stored as document arrays (e.g., an L of M on DynamoDB).
    • Vec<scalar> model fields are stored as text[] on PostgreSQL, JSON on MySQL/SQLite, and a List on DynamoDB.

    Capabilities: Document storage is gated by Capability::document_collections. All four in-tree backends currently support this capability.

  7. Understand Toasty's database operation philosophy

    main

    Toasty is designed to preserve the native behavior of different database backends rather than forcing a single normalized behavior. This results in two types of query methods:

    1. Pass-through methods: These maintain the backend's native behavior. For example, .like() uses the database's own LIKE implementation, meaning case sensitivity will vary depending on whether you are using SQLite, PostgreSQL, or MySQL.
    2. Backend-specific methods: Methods that map to an operator unique to a specific backend are only available on that backend. For example, .ilike() is available for PostgreSQL (mapping to ILIKE) but is rejected on other backends rather than being emulated.

    Toasty only provides a method across all backends if it can guarantee identical semantics everywhere. For example, .starts_with() is available on all backends because it can be expressed with consistent meaning (e.g., DynamoDB's begins_with, PostgreSQL's ^@, SQLite's GLOB, etc.).

  8. Understand the Toasty Execution Model

    main

    The execution phase is the interpreter that runs the compiled program. It follows a sequential loop to process actions:

    1. Initialize variable storage: Sets up the environment for the program.
    2. Action Loop: For every action in the sequence:
      • Loads input data from existing variables.
      • Performs the operation (either a database query via a driver or an in-memory transformation).
      • Stores the resulting data into an output variable.
    3. Return Result: The engine returns the value of the final variable (the output of the last action) to the user.

    To manage memory, the engine tracks variable references. A variable is dropped and its memory freed once the last downstream action that references it has completed.

  9. Understand Atomic Batch Guarantees in Toasty

    main

    Toasty provides atomicity guarantees for batch operations across all supported databases. When using toasty::batch(), create_many(), or performing cascading writes, all operations will either succeed together or fail together.

    Key Behaviors:

    • DynamoDB: Uses TransactWrite to ensure atomicity. A batch that exceeds DynamoDB's limits (100 actions or 4 MB) will return Error::batch_too_large instead of splitting the batch.
    • SQL Backends: Use BEGIN/COMMIT blocks to ensure atomicity.
    • Unsupported Features: If a driver does not support atomic multi-write operations, Toasty returns Error::unsupported_feature when a multi-write batch is attempted.

  10. The Toasty Compilation Pipeline

    main

    Toasty transforms user queries through a multi-phase pipeline to reach an executable form:

    1. Normalization: Expands implicit application-level semantics into explicit AST nodes.
    2. Verification: Validates the normalized statement structure.
    3. Simplification: Optimizes and normalizes the AST (e.g., rewriting associations).
    4. Lowering: Converts the AST into HIR (High-level Intermediate Representation) for dependency analysis.
    5. Planning: Builds a MIR (Middle-level Intermediate Representation) operation graph (a DAG).
    6. Execution Planning: Converts the MIR into a concrete sequence of actions with variable bindings.
    7. Execution: Runs the actions against the database driver.
    8. Result Stream: Returns the final results.

  11. Understand the Toasty Type System Boundaries

    main

    Toasty operates using two distinct type systems to balance developer ergonomics with runtime performance:

    1. Rust-Level Type System (Compile-Time): Uses Rust's generics and traits to provide type-safe model and field access. This prevents common errors like referencing a field that doesn't exist on a specific model or comparing incompatible types (e.g., comparing a String field to a u64 value).

    2. Query Engine Type System (Runtime): Once a statement is passed to db.exec(), the Rust generics are erased. The engine then uses stmt::Type to track the type of values evaluated by statements at runtime, transitioning from high-level Type::Model representations to structural Type::Record representations during the lowering phase.

  12. Understand the GitHub Labeling Scheme

    main

    Toasty uses a prefixed labeling system to categorize issues and pull requests. This allows maintainers and contributors to filter work by type, area, priority, and workflow state.

    Labeling Rules:

    • Each issue must have exactly one C- (Category) label.
    • Once triaged, each issue must have exactly one P- (Priority) label.
    • A- (Area), I- (Impact), and S- (Status) labels are applied as they fit.