DOCSEARCHbyAlgolia
Submit Docs

Malloy

repository·main·Indexed 25 days ago

https://github.com/malloydata/malloy

An open-source semantic modeling and query language built on top of SQL for defining complex data relationships and transformations. Malloy supports connections to BigQuery, Postgres, DuckDB, and Trino. The ecosystem includes a query builder for programmatic AST manipulation, a filter API for specialized data types, a web component renderer for visualizations, and syntax highlighting assets.

Tokens
88.6K
Snippets
150
Records
459
Agent score
82%

What's inside malloy

  1. What is Malloy?

    main

    Malloy is an open-source language designed for describing data relationships and transformations. It serves two primary purposes:

    1. Semantic Modeling Language: Used to define the structure and relationships within your data.
    2. Querying Language: Used to execute queries against relational databases.

    Malloy currently supports connections to BigQuery, Postgres, and DuckDB. To facilitate development, a Visual Studio Code extension is available for building models, querying data, and creating visualizations.

  2. What is Malloy?

    main

    Malloy is a semantic modeling and query language designed to describe data relationships and transformations. It acts as a layer above existing SQL engines, allowing users to build complex data models that are then translated into optimized SQL for execution.

    Key capabilities include:

    • Semantic Modeling: Define joins, dimensions, and measures as part of a data source rather than just in a query.
    • Multi-Database Support: Works with BigQuery, Snowflake, PostgreSQL, MySQL, Trino, Presto, and DuckDB.
    • Symmetric Aggregates: Automatically handles joined data to prevent double-counting during aggregation.
    • Nested Data Access: Uses a consistent dotted-path notation to access data, whether it is stored as nested records/arrays or in separate normalized tables.

  3. Overview of the malloy package structure

    main

    The malloy package contains the core implementation of the Malloy language, including the compiler, translator, and runtime. The package is organized into the following functional areas:

    • src/lang/: The Translator (Parse tree $\rightarrow$ AST $\rightarrow$ IR).
    • src/model/: The Compiler (IR $\rightarrow$ SQL).
    • src/dialect/: Database-specific SQL generation logic.
    • src/api/: The API layers, with src/api/foundation/ containing the public API classes.
    • src/connection/: The Connection registry for managing database connections.
    packages/malloy/
├── src/
│   ├── lang/              # Translator: Parse tree → AST → IR
│   ├── model/             # Compiler: IR → SQL
│   ├── dialect/           # Database-specific SQL generation
│   ├── api/               # API layers
│   │   └── foundation/    # Public API classes
│   └── connection/        # Connection registry

  4. Use @malloydata/malloy to build JavaScript applications

    main
    The @malloydata/malloy package allows you to integrate the Malloy language into your JavaScript products, data applications, or websites. It acts as a compiler that translates Malloy data models and queries into SQL and metadata. This SQL can then be executed against a relational database to retrieve and process data.

  5. Use the Malloy Foundation API

    main

    The Foundation API is the primary object-oriented interface for interacting with Malloy programmatically. It is located in packages/malloy/src/api/foundation/ and provides a fluent way to manage models, queries, and results.

    Key abstractions include:

    • Runtime: The main entry point. It manages connections, URL readers, and manifests. Use it to load models and run queries.
    • Model: A wrapper around the model definition (IR.ModelDef). Provides access to explores and queries. Use getContent(name) for safe lookups.
    • Explore: A wrapper around a source definition (IR.SourceDef), used for field introspection and schema access.
    • PreparedQuery: A query that is linked to a Model. It defers SQL generation until execution.
    • PreparedResult: The output of a prepared query, containing the generated SQL and schema.
    • Result: The final query result containing data, schema, and metadata.

  7. Use Malloy Syntax Highlighting for various Malloy dialects

    main

    The malloy-syntax-highlight package provides syntax highlighting assets for different Malloy dialects. This ensures consistent highlighting across different tools and integrations. Supported dialects include:

    • Standard Malloy syntax (used in .malloy files)
    • Malloy notebook format (.malloynb)
    • Malloy SQL (.malloysql)

  9. How the Malloy Render plugin system works

    main

    The Malloy Render plugin system allows you to extend visualization capabilities by creating custom renderers for specific field types or data patterns. The architecture relies on three main components:

    1. Plugin Factories: Responsible for matching fields to a plugin and instantiating them.
    2. Plugin Instances: Handle the actual rendering logic and data processing.
    3. Plugin Registry: Manages the lifecycle and availability of plugins.

    Plugins are typically triggered by adding tags to fields in Malloy code (e.g., dimension: status # my_plugin).

    source: users is table('users') {
  dimension: 
    status # my_plugin
    age # advanced_viz { "max_value": 100 }
}

  10. Intermediate Representation (IR) Overview

    main

    The Intermediate Representation (IR) is the core data format used by the Malloy compiler. It is composed of plain data structures (not class instances), making it fully serializable. This allows IR to be:

    • Cached between compilations
    • Transmitted over a network
    • Persisted to disk
    • Reused without re-parsing
    • Processed by different compiler versions

    Key IR types include:

    • ModelDef: The complete model definition (sources and queries).
    • SourceDef: A data source (table or derived table) with its schema.
    • StructDef: Schema definition for structured data (records, arrays, tables, etc.).
    • Query: A complete query containing a source and an operation pipeline.
    • Expr: A tree of expression nodes representing computations (arithmetic, logical, aggregate, etc.).

  11. How the Malloy compilation architecture works

    main

    The Malloy compiler uses a two-phase architecture to transform high-level Malloy code into executable SQL:

    1. Translator (packages/malloy/src/lang/):

      • Parses Malloy source code using an ANTLR-generated parser.
      • Generates an Abstract Syntax Tree (AST).
      • Transforms the AST into an Intermediate Representation (IR), which is a serializable data format describing the full semantic model.

    2. Compiler (packages/malloy/src/model/):

      • Consumes the IR.
      • Translates the IR into database-specific SQL queries.
      • Produces the SQL along with necessary metadata to render results using Malloy semantics.

  12. Manage Node.js runtime versioning

    main

    The project uses a .node-version file to pin the Node.js runtime to 24.16.0.

    Reason for the hold: Node.js 24.17.0 introduced a http.Agent keep-alive socket-reuse regression. This causes intermittent ERR_STREAM_PREMATURE_CLOSE errors when fetching from Google's OAuth endpoints (used by BigQuery).

    Management Strategy:

    • The .node-version file is the single authoritative source for the runtime version.
    • CI workflows use actions/setup-node with node-version-file: '.node-version' to respect this pin.
    • The project stays on the Node 24 major until it reaches its end-of-life (LTS), rather than chasing every minor/major release.