DOCSEARCHbyAlgolia
Submit Docs

Feldera

website·Indexed 33 days ago

https://www.feldera.com

Feldera performs incremental view maintenance (IVM) for arbitrary SQL programs, maintaining SQL views over changing data by processing only the changes.

Tokens
1.1K
Snippets
1
Records
10
Agent score
20%

What's inside Feldera

  1. Overview of Feldera Incremental View Maintenance

    Feldera is an open-source engine that performs incremental view maintenance (IVM) for arbitrary SQL programs. Unlike traditional databases that may require full recomputation, Feldera maintains SQL views over changing data by processing only the delta (changes), making compute proportional to the size of the input changes (O(Δ)). It supports full SQL syntax and semantics incrementally, including joins, aggregates, correlated subqueries, window functions, recursive queries, and deeply nested view hierarchies. It is built on DBSP, a formally verified mathematical framework for incremental computation on data streams.

  2. Implement real-time data architectures with Feldera

    Feldera is designed for several high-performance use cases:

    • Real-time Medallion Architecture: Converting batch-based medallion layers into always-fresh incremental pipelines.
    • Fraud Detection: Using time-windowed aggregations and joins for transaction scoring.
    • Fine-Grained Authorization: Implementing permission graphs and reachability via recursive SQL.
    • Batch Migration: Migrating Spark SQL batch jobs to incremental pipelines to reduce latency.

  3. Implement a fresh Medallion Architecture using Feldera

    To keep silver and gold layers of a medallion architecture fresh without scheduled batch rebuilds, declare the silver and gold transformations as SQL views in Feldera. Feldera attaches to sources like Delta Lake and Iceberg, maintains the views incrementally as bronze data arrives, and writes the results back to the lakehouse. This ensures the lakehouse remains the source of truth while updates occur in milliseconds.

  4. Maintain SQL views incrementally with Feldera

    Feldera is an incremental compute engine (not a database) that performs Incremental View Maintenance (IVM) for arbitrary SQL programs. Instead of full refreshes, it processes only the delta (changed rows) to update materialized views. This approach is based on the formally verified DBSP theory, ensuring results are consistent with a full batch recomputation.

    Supported SQL features for incremental maintenance include:

    • Joins and outer joins
    • Aggregates and GROUP BY
    • Window functions
    • Correlated subqueries
    • Recursive and mutually recursive views (using DECLARE RECURSIVE VIEW syntax)

  5. Use Feldera Agent Skill tools

    After installing the feldera-skills package, the following tools are available to the AI agent:

    • install-fda: Install the Feldera environment.
    • install-feldera: Install Feldera.
    • deploy-pipeline: Deploy a pipeline from a SQL file.
    • felderize: Translate Spark SQL into Feldera SQL.
    • feldera-docs: Search the Feldera documentation.

  6. Install and manage Feldera using the fda CLI

    The fda command-line interface is used for managing Feldera pipelines. For local development, installing Feldera via Docker is the preferred method.

  7. Install Feldera Agent Skills for AI coding agents

    Feldera provides a set of installable agent skills (following the Agent Skills Open Standard) that allow AI coding agents—such as Claude Code, Cursor, and GitHub Copilot—to set up and operate Feldera directly. These skills include tools for installation, pipeline deployment, SQL translation, and documentation search.
    npx skills add feldera/feldera-skills

  8. Manage Feldera pipelines via Python SDK or REST API

    Developers can define and control Feldera pipelines programmatically using the following interfaces:

    • Python SDK: A library to define and control pipelines from Python code.
    • REST API: An HTTP API used for pipeline management, data ingress, and view egress.

  9. Connect Feldera pipelines to external data sources and sinks

    Feldera pipelines use connectors to ingest data from heterogeneous sources and write results to various sinks.

    Supported Input Connectors:

    • Kafka
    • Debezium (for CDC streams like MySQL)
    • PostgreSQL
    • Delta Lake
    • Apache Iceberg
    • AWS S3 (batch ingest)
    • HTTP (push/pull)

    Supported Output Connectors:

    • Kafka
    • Delta Lake
    • PostgreSQL
    • HTTP endpoints

    Supported data formats include JSON, Avro, CSV, and Parquet.

  10. Use Feldera SQL for incremental computations

    Feldera supports a wide range of SQL features for incremental processing:

    • Standard SQL: Joins, aggregates, group by, and window functions.
    • Recursive Queries: Used for incremental graph and hierarchy computations (mutually-recursive queries).
    • Time-Series Extensions: Support for tumbling, hopping, and session windows for event-time processing.
    • User-Defined Functions (UDFs): Custom functions can be declared using CREATE FUNCTION.
    • Ad-hoc Queries: On-demand batch queries can be used to inspect the state of a running pipeline.