DOCSEARCHbyAlgolia
Submit Docs

Materialize Documentation

repository·main·Indexed 27 days ago

https://github.com/materializeinc/materialize

A real-time data integration platform using SQL to create and maintain consistent, incrementally updated views of streaming data. It features a PostgreSQL-compatible interface designed for operational dashboards, AI/RAG pipelines, and real-time data products. Documentation includes guides for the Materialize Console, local development with Docker, TypeScript type generation from the system catalog, and platform architecture.

Tokens
229.2K
Snippets
547
Records
1.2K
Agent score
91%

What's inside Materialize

  1. Overview of the COMPUTE layer in Materialize

    main

    The COMPUTE layer is a core component of the Materialize architecture responsible for managing the lifecycle and execution of data processing. Its primary responsibilities include:

    • Dataflow execution: Running the logic defined by your SQL queries.
    • Optimization: Improving query performance and execution plans.
    • Horizontal scaling: Managing the distribution of workloads across multiple nodes.
    • Active replication: Ensuring data availability and consistency through replication mechanisms.

  2. Overview of mzbuild

    main

    mzbuild is a build and orchestration system for Docker containers. It provides two primary interfaces for developers:

    • mzcompose: A layer on top of Docker Compose that allows orchestrating services via Python scripts. It automatically manages image acquisition (downloading from Docker Hub or building locally based on input changes) to keep development cycles fast.
    • mzimage: A lower-level command for inspecting and exercising fine-grained control over the build process of images within the repository.

    From the repository root, these tools are available as bin/mzcompose and bin/mzimage. If a directory contains an mzcompose.py file, a convenience script ./mzcompose is generated for use within that directory.

  3. Overview of mz-deploy capabilities

    main

    mz-deploy is a SQL-native CLI designed to manage Materialize deployments using plain .sql files stored in a git repository.

    Key Features:

    • Git-based Workflow: Organize one object per .sql file, structured by database and schema. mz-deploy tracks dependencies and performs diffs against the live environment to deploy only necessary changes.
    • Offline Type-Checking: Use mz-deploy compile to type-check SQL statements against dependency schemas locally without requiring a live database connection.
    • Unit Testing: Use inline unit tests to mock dependencies and verify view logic with deterministic inputs.
    • Zero-Downtime Deployments: Changes are deployed to isolated staging schemas. Once materialized views finish computing, an atomic swap switches traffic to the new version, ensuring running queries are not interrupted.
    • Converged Infrastructure: Durable objects like secrets, connections, sources, and tables are converged in place (similar to Terraform).

  4. Overview of mz-deploy CLI

    main
    The mz-deploy crate provides a CLI tool designed for developing and deploying Materialize projects. Unlike traditional migration tools or dbt, mz-deploy uses a declarative model where the tool derives the necessary DDL to reach a desired state, rather than requiring users to write ordered migration scripts. It manages the complete lifecycle of a project, including setup, application, staging, promotion, and development workflows.

  5. Overview of Testdrive framework

    main

    Testdrive is an "expected vs. actual output" testing framework used for Materialize system tests. It uses files with a .td extension to specify both actions (like ingesting data into Kafka) and SQL queries, alongside their expected results.

    Key Characteristics:

    • Action Commands ($): Used for external system interactions (e.g., $ kafka-ingest ...).
    • SQL Commands (>): Executes a SQL command and verifies the output. These are retried until a timeout occurs if the output doesn't match.
    • Failure Commands (!): Executes a SQL command and expects it to fail.
    • Best For: Testing sources, sinks, external system interactions, catalog object interactions, and PostgreSQL wire protocol serialization.
    • Not Recommended For: Tests with many EXPLAIN statements or tests where query plans change frequently.

  6. Overview of Self-Managed Materialize Deployments

    main

    Self-Managed Materialize allows you to deploy Materialize within your own infrastructure, as opposed to using the fully managed Materialize Cloud service.

    Deployments on Kubernetes are composed of several layers of components that interact to provide a complete database environment. Understanding these layers is necessary for successful deployment, management, and troubleshooting.

  7. Overview of "scratch" infrastructure

    main

    The "scratch" AWS account provides on-demand infrastructure for quick tests, benchmarks, and short-lived side projects.

    Important Limitations:

    • Machines are automatically deleted after 36 hours.
    • It is NOT intended for long-term or mission-critical infrastructure.
    • For long-term environments, use the --max-age-days option when creating instances via bin/scratch.

    Access:

    • AWS Console: Use the Materialize SSO app.
    • CLI/API: Follow the internal i2 repository instructions for AWS access.

  8. Overview of the materialize-agent MCP Server

    main

    Starting in v26.24, Materialize provides a built-in materialize-agent Model Context Protocol (MCP) server located at /api/mcp/agent (port 6876). This server allows AI agents (such as Claude Code, Claude Cowork, or Cursor) to discover and query business-facing data products using natural language or SQL.

    Key Features:

    • No sidecar process or external server required.
    • Supports querying via natural language (e.g., "What's the total_revenue for product 42?") or direct SQL (e.g., "SELECT * FROM mcp_product_performance LIMIT 5;").
    • Connects via HTTP.

  9. Overview of mz-deploy CLI

    main

    mz-deploy is a dedicated CLI designed for developing and deploying Materialize projects. It addresses the limitations of using general-purpose tools like dbt or Terraform by providing a specialized project model, an offline compiler/typechecker, a unit test harness, and a managed blue/green deployment lifecycle.

    Key capabilities include:

    • Project-based management: Describe a Materialize environment as a checked-in directory of SQL files.
    • Offline Validation: Uses Materialize's real parser and planner to catch SQL errors (typos, type mismatches) before deployment without needing a live server connection.
    • Zero-Downtime Deployments: Implements a blue/green pattern with atomic cutover, ensuring production objects remain queryable.
    • Crash-Safe Operations: Deployment steps are resumable; killing the tool mid-process allows rerunning the command to resume correctly without data loss.
    • Unit Testing: Allows testing view logic against inline fixture data locally or in CI without a remote environment.
    • Replacement Views: Enables updating materialized views in place so downstream consumers remain untouched.
    • Concurrency Safety: Designed to handle multiple engineers deploying simultaneously through metadata design rather than an external lock service.

  10. Overview of the ADAPTER layer in Materialize

    main

    The ADAPTER layer is a core component of the Materialize database architecture. It manages the interface between users/applications and the underlying database engine. Its primary responsibilities include:

    • SQL interface: Handling incoming SQL queries.
    • Transactions: Managing transactional integrity and execution.
    • Coordination and timelines: Orchestrating system coordination and managing data timelines.
    • Version upgrades: Facilitating database version upgrades.

  11. Overview of the Materialize SQL parser

    main
    The Materialize SQL parser is a specialized fork of sqlparser-rs (including patches from nickolay/sqlparser-rs). Unlike the upstream version, this parser is optimized specifically for the Materialize SQL dialect. It has intentionally diverged from upstream compatibility to allow for large-scale refactors and to support Materialize-specific syntax rather than general-purpose dialects like MySQL or MSSQL.

  12. Overview of the Materialize Developer MCP Server

    main

    Materialize includes a built-in Model Context Protocol (MCP) server specifically designed for troubleshooting and observability. This server is provided directly by Materialize, meaning no external server or sidecar process is required to run it.

    Endpoint Details:

    • API Path: /api/mcp/developer
    • Port: 6876

    You can connect MCP-compatible clients (such as Claude Code, Claude Desktop/Cowork, or Cursor) to this server to interact with your Materialize system.

    Capabilities:

    • System Troubleshooting: Ask questions about the Materialize system (e.g., investigating stale materialized views or cluster memory usage).
    • Object Querying: Run queries on your objects (available starting in v26.30), such as SELECT * from a view or examining memory usage and skew for specific objects.