DOCSEARCHbyAlgolia
Submit Docs

Substrait Documentation

repository·main·Indexed 23 days ago

https://github.com/substrait-io/substrait

Substrait is a cross-language specification providing a standardized, independent description of data compute operations. It defines a formal specification, human-readable text representations, and a compact binary representation for sharing compute plans across different systems. Complementary to Apache Arrow, Substrait focuses on the operations performed on columnar data, serving as an open format for logical and physical query plans generated by SQL systems.

Tokens
41.7K
Snippets
46
Records
187
Agent score
68%

What's inside Substrait

  1. Overview of Substrait

    main

    Substrait is a project designed to produce an independent description of data compute operations. It provides a standardized way to describe compute plans that can be shared across different systems. The project consists of three primary components:

    1. A formal specification: The underlying rules and definitions for compute operations.
    2. A human-readable text representation: For inspecting and debugging compute plans.
    3. A compact cross-language binary representation: For efficient, high-performance exchange of compute plans between different languages and systems.

  2. Overview of the Substrait Specification components

    main

    The Substrait specification defines a standardized way to describe compute plans, including types, expressions, and relations. The specification is divided into implemented components and components that are currently designed but not yet implemented.

    Implemented Components

    Types

    • Simple Types: Basic scalar types like integers and doubles.
    • Compound Types: Types beyond scalars, including fixed-length, numeric, structs, maps, and lists.
    • Type Variations: Physical variations to base types.
    • User Defined Types: Extensions for specific IR (Intermediate Representation) producers or consumers.

    Expressions

    • Field References: Expressions used to identify specific portions of a record.
    • Scalar Functions: Function specifications including arguments, variadic support, and output type derivation. A canonical list of functions is maintained in extensions/scalar_functions.yaml.
    • Specialized Record Expressions: Non-function paradigm expressions like if/then/else or switch statements.
    • Aggregate Functions: Operations that collapse multiple records into a single value (e.g., SUM, COUNT).
    • Window Functions: Functions relating a record to a set of encompassing records (e.g., RANK, NTILE).
    • User Defined Functions (UDFs): Reusable named functions registered via external means (e.g., dropping a file in a directory).
    • Embedded Functions: Function implementations embedded directly within the plan, common in data science workflows.

    Relations

    • Relation Basics: Core concepts of relational algebra and record properties.
    • Logical Relations: Common relational operations like project, join, and aggregation.

    Serialization

    • Text Serialization: A human-readable representation of the plan.
    • Binary Serialization: A high-performance, compact binary representation for machine use.

    Designed but Not Implemented Components

    • Table Functions: Functions converting input values into 0..N output records (e.g., explode).
    • User Defined Relations: Reusable relational operations customized to a platform.
    • Embedded Relations: Relational operations where plans contain direct execution "machine code".
    • Physical Relations: Specific execution sub-variations of logical operations (e.g., hash join, merge join, nested loop join).

  4. Directory structure of Substrait documentation examples

    main

    The examples/ directory contains standalone files used throughout the Substrait documentation. These files are kept separate from the Markdown documentation to allow for automated schema validation in CI/CD.

    • extensions/: Contains examples for extension functions (e.g., any types).
    • types/: Contains examples for user-defined types.

  6. Identify Substrait-compatible query engines and tools

    main

    Substrait is adopted by various open-source projects that act as either producers (creating Substrait plans) or consumers (executing Substrait plans). Use the following list to find tools compatible with your workflow:

    Consumers (Engines that execute Substrait plans)

    • Acero: A query execution engine within the Apache Arrow C++ library.
    • ADBC (Arrow Database Connectivity): An API for Arrow-based database access that accepts Substrait plans.
    • Arrow Flight SQL: A client-server protocol that allows clients to send queries as Substrait plans.
    • Velox: A unified execution engine designed to accelerate data management systems.
    • DuckDB: An in-process SQL OLAP database that can consume plans via the Substrait Community Extension.

    Producers (Tools that generate Substrait plans)

    • DataFusion: A Rust-based query framework that can convert logical plans to Substrait plans (available via Python bindings).
    • Gluten: A plugin for Apache Spark that converts Spark physical plans into Substrait plans.
    • Ibis: A Python data wrangling library that includes a Substrait producer to enable support for consuming engines.
    • Substrait R Interface: An R package that allows constructing Substrait plans from R, including a dplyr backend.

    Bi-directional (Both Producer and Consumer)

    • DataFusion: Can convert plans to and from Substrait.
    • DuckDB: Can produce and consume plans via the Substrait Community Extension.

  8. What is an Extended Expression and when to use it

    main

    An ExtendedExpression is an alternative to using a full Plan for expression-level protocols. It is specifically designed for expression-only evaluations, such as those occurring within Filter, Project, or Aggregation relational operators.

    Unlike the standard Expression message, an ExtendedExpression is self-contained and provides the necessary context for evaluation without requiring a full relational plan. To achieve this, it includes:

    • Version: Basic protocol versioning.
    • Input and output data schema: Describes the input data (name, type, nullability, and layout) and the name of the output data.
    • Referred expressions: The actual computation logic (using Expression or AggregateFunction).
    • Function extensions: To support custom function signatures.

  9. What is the unbound type and how should it be handled?

    main

    The unbound type is a placeholder used in partially bound plans. It allows a producer to serialize the shape of a plan (relations, field names, and ordering) before the concrete types of those fields are known. A downstream binder is responsible for replacing unbound with concrete types using schema or catalog information.

    Key characteristics of unbound:

    • Not a wildcard: It is distinct from any. While any is a function-signature wildcard for overload matching, unbound is a plan-level placeholder.
    • No properties: It carries no nullability and accepts no parameters; these are determined during binding.
    • No literals: There is no literal representation for the unbound type.
    • Non-executable: Partially bound plans containing unbound are not executable, and Substrait defines no runtime semantics for its values.

    Developer Guidance: If your system does not support partially bound plans, you should reject any plan containing the unbound type.

  10. What is Substrait?

    main
    Substrait is a cross-language specification for data compute operations. While projects like Apache Arrow focus on a standardized memory representation of columnar data, Substrait focuses on the operations performed on that data. It defines a set of common operations, their semantics, and their behavior unambiguously, providing extension points and serialized representations for these plans.

  11. Use If Expressions with short-circuiting behavior

    main

    An if expression consists of one if clause, zero or more else if clauses, and an else clause.

    Requirements:

    • All return expressions must be of the identical type.

    Short-circuiting: if expressions are expected to logically short-circuit on a positive outcome. This means that a skipped else or else if expression cannot cause an error. For example, a failing cast in a skipped branch should not throw an error.

    if 'value' = 'value' then 0
else cast('hello' as integer)

  12. Understand the Substrait volunteer roles

    main

    The Substrait project is managed by volunteers categorized into four distinct roles based on their level of involvement and access:

    • User: Someone who uses Substrait and provides feedback via bug reports or feature suggestions.
    • Contributor: A user who actively contributes code, documentation, or patches and participates in developer mailing list discussions. Contributors may be granted write access to specific -contrib repositories via PMC consensus.
    • Committer: A developer with write access to all core and -contrib repositories who has signed a Contributor License Agreement (CLA). They make short-term decisions for the project.
    • PMC Member (Project Management Committee): An elected committer responsible for the long-term health of the project, managing the governance model, and casting binding votes on community-related decisions.