Wren Engine Documentation

repository·main·Indexed 20 days ago

https://github.com/canner/wren-engine

An open-source semantic and execution foundation that provides AI agents with business context by transforming raw data into a governed layer of models, metrics, and relationships using Modeling Definition Language (MDL). The project includes the wren-engine-server (v0.25.0), a CLI tool, and an Ibis Server module for executing SQL queries via FastAPI, ibis, and sqlglot.

Tokens
109.3K
Snippets
341
Records
511
Agent score
70%

What's inside Wren Engine

  1. Overview of the Wren Core Module

    main
    The Wren Core module is the semantic core of the Wren engine. It is primarily used for SQL planning within the ibis-server API v3. For Python users, the functionality is exposed via the wren-core-py module, which is also utilized by the ibis-server.
  2. Overview of Ibis Server Module

    main

    The Ibis Server Module is the API server for Wren Engine, built on top of FastAPI. It provides APIs for executing SQL queries. The query lifecycle involves:

    1. Planning the SQL query via wren-core.
    2. Transpiling the query using sqlglot.
    3. Executing the query via ibis against the target database.

    Note: This module is deprecated. Wren Engine is migrating to a CLI tool with similar capabilities. For the new tool, refer to the wren package.

  3. Interact with Wren MCP via HTTP JSON-RPC

    main

    The wren-http-api allows you to interact with the Wren MCP server using plain HTTP JSON-RPC 2.0 requests. This is useful when your client cannot use the standard MCP SDK (e.g., custom HTTP clients, shell scripts, or environments like OpenClaw).

    Key Details:

    • Protocol: JSON-RPC 2.0 over HTTP POST.
    • Base URL: http://localhost:9000/mcp (default Docker setup).
    • Required Headers:
      • Content-Type: application/json
      • Accept: application/json, text/event-stream
    • Transport: Uses streamable-http transport, meaning responses arrive as Server-Sent Events (SSE).
  4. What is a Wren MDL Project?

    main

    A Wren MDL project is a directory of YAML files used to manage Wren Model Definition Language (MDL) manifests in a human-readable and version-control friendly format, similar to how dbt projects operate.

    Instead of managing a single large JSON file, you split the manifest into several files: one for project metadata, one per model, one for all relationships, and one for all views. This structure allows for easier code reviews and granular version control.

    Key distinction: The YAML files use snake_case for field names to improve readability, while the compiled output (target/mdl.json) uses camelCase to match the wire format expected by the engine.

  5. What is a View in Wren Engine

    main

    A View is a named SQL query stored in the MDL (Model Definition Language). It acts as a virtual table that clients can query by name.

    Key characteristics:

    • Inlining: The engine inlines the statement SQL before execution.
    • Schema Inference: Unlike a Model, a View does not require explicit column declarations; its schema is inferred from the statement at query time.
    • Recursive Expansion: Views can reference other views. The engine expands these references recursively before resolving any underlying models.

    Use a View for pre-built queries such as dashboards, saved filters, or cross-model aggregations that you want to expose as a named table.

  6. What is Modeling Definition Language (MDL)?

    main

    Modeling Definition Language (MDL) is the structured, machine-readable language used by Wren Engine to describe business data. Instead of exposing raw database tables and columns, MDL provides a logical layer that defines how data should be interpreted and used.

    MDL allows you to define:

    • Models: Logical representations that reference physical tables or query results.
    • Columns and Expressions: Specific field definitions and their underlying logic.
    • Relationships: How different models connect to one another.
    • Calculated Fields and Metrics: Reusable business calculations and aggregations.
    • Views: Analytical views built on top of modeled datasets.

    By using MDL, you provide Wren Engine with a consistent business context, which is essential for reliable query generation and AI agent reasoning.

  7. What is a Model in Wren MDL?

    main
    A Model is the fundamental building block of Wren MDL. It acts as a semantic layer that maps a physical database table or a SQL expression to a named entity. This allows AI agents and SQL clients to query data using intuitive, business-friendly names (e.g., SELECT * FROM customers) instead of complex physical paths. Models define the exposed columns, their data types, and how they relate to other models.
  8. What is Wren Engine and how does it work?

    main

    Wren Engine is an open context engine designed to provide AI agents with a semantic, governed layer for business data. Instead of agents interacting with raw database tables, Wren Engine allows them to reason over business concepts.

    Core Workflow

    1. Modeling: Describe your business domain using Wren's semantic model and Modeling Definition Language (MDL).
    2. Context Capture: Wren Engine captures models, metrics, relationships, and access rules.
    3. Intent Analysis: The engine analyzes natural language intent and plans correct queries across underlying data sources.
    4. Interaction: MCP clients and AI agents interact with this context through a clean interface.

    Key Benefits for Agents

    • Understand models instead of raw tables.
    • Use trusted metrics instead of inventing SQL.
    • Follow relationships instead of guessing joins.
    • Respect governance instead of bypassing it.
  9. How CTE-Based Modeling works in Wren SQL

    main

    Wren Engine uses a rewrite pipeline to transform your SQL. It injects Common Table Expressions (CTEs) that expand each MDL (Model Definition Language) model into its underlying database query.

    The Rewrite Pipeline:

    1. Parse & Qualify: Uses sqlglot to parse your SQL and qualify column references.
    2. Identify: Detects which models and columns are referenced in your query.
    3. Expand: For each model, wren-core expands the model definition into a CTE.
    4. Inject: The model CTEs are injected into your query.
    5. Output: The final SQL is produced in the target database dialect.

    Example Transformation: If you have a model orders backed by table public.orders with columns o_orderkey, o_custkey, and o_totalprice:

    Your Input:

    SELECT o_custkey, SUM(o_totalprice) FROM orders GROUP BY 1

    Engine Output (via dry-plan):

    WITH "orders" AS (
      SELECT "public"."orders"."o_orderkey",
             "public"."orders"."o_custkey",
             "public"."orders"."o_totalprice"
      FROM "public"."orders"
    )
    SELECT o_custkey, SUM(o_totalprice) FROM orders GROUP BY 1

    The CTE named "orders" shadows the model name, allowing the rest of your SQL to run against the CTE as if it were a standard table.

    -- You write:
    SELECT o_custkey, SUM(o_totalprice) FROM orders GROUP BY 1
    
    -- Engine produces (via dry-plan):
    WITH "orders" AS (
      SELECT "public"."orders"."o_orderkey",
             "public"."orders"."o_custkey",
             "public"."orders"."o_totalprice"
      FROM "public"."orders"
    )
    SELECT o_custkey, SUM(o_totalprice) FROM orders GROUP BY 1
  10. Use Memory to provide context to AI agents

    main

    The Memory layer is a LanceDB-backed semantic index. While models define what the data looks like, memory helps agents find the right parts of it.

    Use Memory when you need to:

    • Provide relevant schema context to an AI agent per question (avoiding sending the entire schema in every prompt).
    • Store confirmed NL-SQL pairs as few-shot examples for future queries.
    • Improve query accuracy over time as more examples are stored.
  11. MDL Manifest Structure

    main

    The Wren MDL is a JSON manifest that defines the catalog, schema, data source, models (tables), columns, and relationships.

    Key Mapping Rules for Generation:

    • catalog: Use "wren" unless specified otherwise.
    • schema: Use the target schema name (e.g., "public").
    • dataSource: Use the enum value of the data source (e.g., "POSTGRES").
    • tableReference.catalog: Set this to the actual database name, not "wren".
    • models: Each table in the database corresponds to one Model entry.
    • columns: Each column in the table corresponds to one Column entry. Mark primary keys with "isPrimaryKey": true and update the model's primaryKey field.
    • relationships: Define links between models using joinType and a condition string.
    {
      "catalog": "wren",
      "schema": "public",
      "dataSource": "POSTGRES",
      "models": [
        {
          "name": "orders",
          "tableReference": {
            "catalog": "my_db_name",
            "schema": "public",
            "table": "orders"
          },
          "columns": [
            {
              "name": "order_id",
              "type": "INTEGER",
              "isPrimaryKey": true,
              "isCalculated": false,
              "notNull": true
            }
          ],
          "primaryKey": "order_id"
        }
      ],
      "relationships": [
        {
          "name": "orders_customer",
          "models": ["orders", "customers"],
          "joinType": "MANY_TO_ONE",
          "condition": "orders.customer_id = customers.customer_id"
        }
      ]
    }