DOCSEARCHbyAlgolia
Submit Docs

Estuary Flow

repository·master·Indexed 21 days ago

https://github.com/estuary/flow

A Right-Time Data Platform that unifies batch and streaming data processing for capturing, transforming, and materializing data with millisecond latency. The repository includes the Control Plane Agent for API hosting and spec management, an automations framework for fault-tolerant background tasks using the Executor trait, and tools for inspecting catalog statistics via BigTable and Postgres.

Tokens
558.6K
Snippets
1K
Records
2.1K
Agent score
75%

What's inside estuary-flow

  1. Overview of Estuary security and compliance

    master

    Estuary provides several mechanisms to ensure data security and regulatory compliance. Security is managed through technical features, flexible deployment models, and adherence to industry standards.

    Security Features

    Estuary implements a defense-in-depth strategy using:

    • Data encryption: Protection for data both in motion and at rest.
    • Immutable infrastructure: Reducing the attack surface by using non-persistent, reproducible infrastructure.
    • Zero-trust network model: Ensuring every request is authenticated and authorized.
    • Role-based access control (RBAC): Managing user permissions through granular roles.

    Deployment Options

    For organizations requiring higher levels of isolation, Estuary supports:

    • Private deployments: Dedicated environments.
    • Bring Your Own Cloud (BYOC): Deployments where the data plane remains within your own private cloud infrastructure to maintain data sovereignty.

    Compliance Standards

    Estuary maintains compliance with several major regulatory frameworks:

    • Healthcare: HIPAA
    • Privacy: GDPR, CCPA, and CPRA
    • Security Audits: SOC 2 Type II certified

  2. Overview of Estuary security features

    master

    Estuary provides several built-in security capabilities to protect data and infrastructure:

    • Data Encryption: Data is encrypted both in motion and at rest. Connector secrets are automatically encrypted regardless of whether they are configured via the Estuary UI or the flowctl CLI.
    • Immutable Infrastructure: Systems are rebuilt with every update to maintain security and up-to-date configurations.
    • Zero-trust Network Model: All communications are secured using TLS and mutual TLS (mTLS) for internal communications.
    • Role-based access control (RBAC): Provides secure resource sharing through centralized verification of authorizations.

  3. Overview of Estuary Right-Time Data Platform

    master

    Estuary is a data platform designed to unify batch and streaming paradigms, allowing systems to synchronize around datasets with millisecond latency. It enables three primary data operations:

    1. Capture: Ingest data from systems, services, or SaaS into collections. Collections are millisecond-latency datasets stored as JSON files in cloud storage.
    2. Materialize: Project a collection as a view into another system, such as a database, key/value store, Webhook API, or pub/sub service.
    3. Derive: Create new collections by transforming existing ones using stateful stream workflows, joins, and aggregations in real time.

    Estuary can be used in a shared SaaS environment, or via private/BYOC (Bring Your Own Cloud) deployments for enterprise security.

  4. Overview of the Dekaf connector crate

    master
    The dekaf-connector crate provides the specific components of the dekaf system required to perform build-time validations. It is architecturally decoupled from the main dekaf crate to avoid circular dependencies: while the main dekaf crate depends on flow-client, dekaf-connector does not. This allows it to be linked to control-plane-api (via the runtime crate) without creating a dependency cycle.

  5. What is Estuary?

    master

    Estuary is a 'Right-Time Data Platform' designed for unified data movement and transformation. It synchronizes various systems—including SaaS, databases, and streaming services—by storing datasets in the cloud and updating them with millisecond latency.

    Key capabilities include:

    • Change Data Capture (CDC): Moving data from databases (e.g., MySQL) into destinations like PostgreSQL, Snowflake, or Databricks.
    • SaaS Integration: Capturing data from vendors (e.g., HubSpot, Facebook) into an Apache Iceberg lakehouse.
    • Log Processing: Fetching, transforming, and loading logs from CDNs into Elasticsearch or BigQuery.
    • Real-time Analytics Hydration: Feeding real-time systems like Tinybird or StarTree.
    • Business Event Instrumentation: Making real-time analytics accessible via tools like PostgreSQL or Google Sheets.

    The platform is built on cloud-native streaming infrastructure, a data processing runtime, and an open-source ecosystem of pluggable connectors.

  6. Overview of the Supabase Control Plane

    master

    The Supabase Control Plane is the management layer for Flow. It is responsible for managing the catalog, user accounts, billing, and platform configuration.

    Its architecture consists of:

    • PostgreSQL database: Serves as the core storage for the catalog and platform state.
    • Edge functions: TypeScript-based serverless functions handling OAuth, billing, and alerts.
    • Authentication: Manages user identity and JWT-based authorization.

  8. Overview of the runtime-sidecar process

    master

    The runtime-sidecar is a production sidecar process used in the runtime-v2 architecture. It is intended to run one instance per reactor machine, supervised by systemd. The sidecar hosts two primary gRPC services on a fixed fleet-wide port:

    1. Shuffle Leader (runtime_next::leader::Service): Handles per-task Join rendezvous and HeadFSM/TailFSM coordination for tasks where shard zero is located on the local machine.
    2. Shuffle (shuffle::Service): Handles Session, Slice, and Log RPCs.

  9. Overview of the PostgreSQL Capture Connector

    master

    The PostgreSQL capture connector uses Change Data Capture (CDC) to continuously stream updates from a PostgreSQL database into Estuary collections. It relies on PostgreSQL's logical replication mechanism to track and capture changes.

    Note: If you are using a managed PostgreSQL instance that does not support logical replication, use the PostgreSQL Batch Connector instead.

  10. What is simd-doc?

    master

    simd-doc is a high-performance JSON parser designed to transcode JSON documents directly into Flow's internal document representations. It uses SIMD instructions (via simdjson) to achieve high throughput, specifically targeting two output formats:

    1. doc::HeapNode: Heap-allocated document nodes.
    2. doc::ArchivedNode: An rkyv-serialized zero-copy document format.

    To ensure robustness, the parser automatically falls back to serde_json for any document exceeding 1MB to avoid performance degradation associated with simdjson on very large inputs.

  11. Overview of Estuary Capture Connectors

    master

    Estuary provides a variety of capture connectors designed to ingest data from databases, SaaS applications, files, streams, and APIs.

    Each connector requires a unique configuration to function correctly. When using the Estuary web app, configuration is handled through the UI. However, if you are developing locally using flowctl, you will need to reference the specific Docker images for each connector to write Data Flow specifications manually.

  12. Use the PostgreSQL Batch Query Connector

    master

    The PostgreSQL Batch Query connector captures data from PostgreSQL databases (tables, views, or custom SQL results) by periodically executing queries and converting them into JSON documents.

    When to use this connector instead of CDC: Use this connector if your PostgreSQL instance does not support logical replication, if you need to capture from a read replica on PostgreSQL <= 15, if you need to capture from database views, or if you want to execute custom/ad-hoc SQL queries.

    Prerequisites:

    • A PostgreSQL database with a user having SELECT permissions on target tables.
    • Network access to the database (direct or via SSH tunnel).
    captures:
  ${PREFIX}/${CAPTURE_NAME}:
    endpoint:
      connector:
        image: ghcr.io/estuary/source-postgres-batch:v1
        config:
          address: host:port
          user: estuary_user
          password: secret
          database: postgres
    bindings:
      - resource:
          name: main_orders
          schema: main
          table: orders
        target: ${PREFIX}/${COLLECTION_NAME}