DOCSEARCHbyAlgolia
Submit Docs

Estuary Documentation

website·Indexed 38 days ago

https://docs.estuary.dev

Documentation for Estuary's data pipeline platform, featuring a comprehensive reference for capture and materialization connectors. Includes guides for integrating data from sources such as Shopify, LinkedIn Ads, Zendesk, and TikTok into Estuary collections, as well as details on Dekaf, Estuary's Kafka-API compatibility layer.

Tokens
304.8K
Snippets
867
Records
2.1K
Agent score
67%

What's inside Estuary

  1. Overview of Zoho CRM capture connector

    The Zoho CRM connector captures data from Zoho CRM modules into Estuary collections. It utilizes Zoho's Bulk API 2.0 for efficient initial data loads and backfills, and the COQL (CRM Object Query Language) API for real-time incremental updates to detect and capture changes since the last sync.

  2. Overview of Estuary Data Flows

    Estuary uses Data Flows to connect data systems, APIs, and storage. A complete end-to-end Data Flow consists of three primary components:

    1. Capture: Ingests data from an external source into Estuary.
    2. Collections: Acts as a real-time data lake in cloud storage where captured data is maintained.
    3. Materialization: Pushes data from Estuary collections to an external destination.

    Optional components include Derivations, which transform data from one or more collections into a new collection.

  3. Overview of Estuary Redaction Strategies

    Redaction in Estuary is a security feature used to protect PII or sensitive data by blocking or hashing fields during write time, before documents are fully captured. This ensures sensitive values are not written to disk or exposed in error messages.

    There are two primary redaction strategies:

    1. Block: Completely removes the field from the document. Use this when downstream systems do not need any information from the field (e.g., removing a Social Security Number).
    2. Hash: Replaces the value with a salted SHA-256 hash. Use this when downstream systems need a consistent identifier for a user or entity but should not have access to the actual PII (e.g., hashing an email for user journey analysis).

  4. Overview of Dekaf materialization connector

    The Dekaf connector materializes Estuary collections as Kafka-compatible messages that can be read by Kafka consumers. It is intended for users who want to consume Estuary data via the Kafka protocol without managing their own Kafka broker. If you have your own Kafka broker, use the Apache Kafka materialization connector instead.

  5. Overview of Estuary materialization connectors

    Estuary provides a wide range of materialization connectors to move data from Estuary to various external destinations. Each connector has specific configuration requirements, properties, and best practices. Currently, all available materialization connectors are developed by Estuary, though third-party open-source connectors may be supported in the future.

  6. Overview of the Gong source connector

    The Gong source connector captures data from the Gong revenue intelligence platform into Estuary collections. It supports the replication of the following data resources via the Gong API: calls, users, scorecards, and scorecard_definitions. By default, each resource is mapped to a separate Estuary collection through a binding.

  7. Overview of Estuary Connectors

    Connectors are Docker images that bridge Estuary with external endpoints for capturing or materializing data. They are dual-licensed under Apache 2.0 or MIT and support both the Airbyte specification and Estuary's low-latency gRPC protocols. Most connectors are available directly within the Estuary web application.

  8. Overview of flowctl subcommands

    flowctl organizes functionality into several top-level subcommands:

    • auth: Manage authentication sessions and provision Estuary roles/users.
    • catalog: Manage active catalog entities (list, pull, test, publish, and delete specifications).
    • collections: Read data from collections to stdout or list journals/fragments.
    • discover: Find available bindings for a specific capture.
    • draft: Alternative workflow for managing specifications.
    • generate: Create stub files and folder structures from a flow.yaml file.
    • logs: Review or follow logs for captures, derivations, and materializations.
    • preview: Run a capture, derivation, or materialization locally to verify output.

  9. Overview of AppsFlyer connector data capture modes

    The AppsFlyer connector captures mobile marketing analytics data into Estuary collections using two distinct modes:

    1. Webhook mode: Provides real-time event capture from AppsFlyer postback webhooks.
    2. Pull API mode: Provides historical and incremental data via AppsFlyer's Aggregate Report APIs.

  11. Overview of the HTTP Ingest connector

    The HTTP Ingest connector captures data from incoming HTTP requests, such as webhook deliveries, and converts them into Estuary collections. Unlike most capture connectors, it does not pull data from a specific system; instead, it accepts any valid JSON objects sent to its public endpoint. It is primarily used for capturing webhooks or testing data ingestion into Estuary.

  12. Overview of AWS EMR Serverless Application requirements

    When using AWS EMR Serverless for compute with the Iceberg connector, the following components must be configured:

    • AWS EMR Serverless Application: The target where the connector submits and monitors jobs.
    • AWS Systems Manager Parameter Store: Used by the connector to securely store catalog credentials, which are then read by EMR jobs.
    • S3 Bucket: Used to stage data that EMR jobs will merge into tables.
    • IAM Execution Role: An IAM role with a specific ARN used to run the EMR jobs.