DOCSEARCHbyAlgolia
Submit Docs

Meltano Documentation

website·Indexed 35 days ago

https://docs.meltano.com

Documentation for Meltano and Meltano Cloud, covering the CLI, plugin configuration, and the Cloud API. Includes guides on managing workspaces, configuring data pipelines via YAML (DataML), using extractors and loaders (Taps and Targets), and integrating with GitHub for configuration management.

Tokens
141.6K
Snippets
692
Records
1.1K
Agent score
67%

What's inside Meltano

  1. Overview of the SurveyMonkey extractor

    The SurveyMonkey extractor syncs survey definitions, distribution data, and individual responses into a data warehouse. It provides data both as raw nested objects and as pre-flattened relational tables to eliminate the need for warehouse-specific SQL to unnest survey or response data.

  2. Overview of Meltano Community Edition

    Meltano Community is the self-hosted, open-source edition of Meltano. It allows developers to install the software locally or on private servers, manage their own infrastructure, and define data pipelines as code. Key capabilities include a CLI-first workflow for managing plugins and settings, access to over 600 connectors via Meltano Hub, and a project-based configuration system where all settings are stored in a version-controlled meltano.yml file.

  3. Overview of tap-google-analytics

    The tap-google-analytics extractor allows teams to sync Google Analytics 4 (GA4) data—including traffic, engagement, locations, devices, pages, and revenue—into a repeatable pipeline. It supports incremental syncs and allows for either a fixed set of default reports or custom report definitions using GA4 dimensions and metrics.

  4. Overview of tap-aptem extractor

    tap-aptem is an extractor for Aptem, an end-to-end apprenticeship and vocational training management platform. It allows training providers to sync learner enrolment, programme delivery, progress tracking (reviews, milestones, tasks, evidence), compliance documents, end-point assessment data, and organization/user administration data into their own reporting systems.

  5. Overview of tap-weatherapi extractor

    The tap-weatherapi extractor connects to the WeatherAPI REST API to sync weather data for one or more configured locations. It supports two primary streams: forecast (daily forecast data, using full-table refresh) and historical (historical daily weather data, using incremental replication based on the date key). Location queries can be performed using city names, US ZIP codes, UK/Canada postcodes, latitude/longitude pairs, airport codes, IP addresses, or Search API IDs.

  6. Overview of target-clickhouse loader

    The target-clickhouse loader allows Meltano to sync extracted data into ClickHouse, a column-oriented OLAP database. It supports append-only, upsert, and overwrite loading methods. In ClickHouse, the database serves as the schema namespace. It can be used as a full data store, meaning extracted data, pipeline state, and dbt transformations all reside in the same ClickHouse instance.

  7. Overview of tap-postgres sync methods

    The tap-postgres connector supports three replication methods that can be set independently per stream, allowing a single pipeline to mix and match methods across different tables:

    1. Full table: Re-extracts the entire table on every run. Best for small or reference tables. No special setup required beyond a standard connection.
    2. Key-based incremental: Extracts only rows added or changed since the last run using a monotonically increasing replication key (e.g., updated_at or an auto-incrementing id). Note: Hard deletes on the source are not captured.
    3. Log-based (CDC): Reads row-level changes (inserts, updates, and deletes) directly from the PostgreSQL write-ahead log (WAL). This is the only method that captures deletes, but it requires server-side configuration.

  8. Overview of tap-rakutenadvertising capabilities

    The tap-rakutenadvertising extractor syncs data from Rakuten Advertising's affiliate marketing platform. It supports three main API groups:

    1. Core API: General data including advertisers, events, partnerships, offers, and coupons.
    2. Advanced Reports API: Financial data such as payment history and advertiser payments.
    3. Reporting Platform: Custom reports built within the Rakuten Advertising UI.

    Replication Methods:

    • Incremental: Supported for transaction-based streams (e.g., events, publisher contributed conversions).
    • Full-table: Used for reference data streams.

  9. Overview of Meltano Cloud Data Stores

    Data stores are used in Meltano Cloud to determine where data imports load data into and where datasets query data from. Every workspace includes a default PostgreSQL data store called 'Warehouse' that is automatically marked as MANAGED and DEFAULT.

    Supported data stores include:

    • PostgreSQL
    • Snowflake
    • BigQuery
    • Microsoft SQL Server
    • ClickHouse
    • MotherDuck

    Note: Custom data stores (those not natively provided by Meltano) support loading data via data import runs, but do not support querying via datasets. If a custom store is set as default, datasets will appear empty.

  10. Overview of Meltano Extractors

    Extractors are pip packages used by meltano run or meltano invoke to pull data from sources such as databases, SaaS APIs, or file formats. Meltano supports Singer taps, which are executables that implement the Singer specification.

  11. Overview of Meltano Mappers

    Mappers (also known as inline stream maps) allow developers to alter data on the fly as it passes through EL pipelines. They are categorized into two levels:

    Stream Level: Modifies the stream itself. Capabilities include:

    • Aliasing: Custom naming for streams downstream.
    • Filtering: Removing records based on user-defined logic.
    • Duplication: Splitting or duplicating streams to send to multiple targets.

    Property (Column) Level: Modifies the contents of the stream. Capabilities include:

    • Aliasing: Renaming columns (e.g., id to customer_id).
    • Transformations: Inline changes like upper-casing or type casting.
    • Exclusions: Removing specific properties.
    • Additions: Creating new properties via expressions.
    • Flattening: Breaking complex nested properties into distinct fields.

    Limitations: Mappers do not support data aggregation, joins between streams, or external API lookups. These tasks should be handled by transformation plugins like dbt after the data lands in the destination.

  12. Overview of Meltano Cloud features

    Meltano Cloud is a fully managed, hosted edition of Meltano designed for production ELT pipelines. It removes the need for infrastructure management by handling deployments, scheduling, credentials, and monitoring. Key characteristics include:

    • Git-backed: Every workspace is linked to a GitHub repository, ensuring pipeline configurations are version-controlled and auditable.
    • Isolation: Workspaces provide isolated environments for teams or projects, each with its own pipelines, data store connections, members, and credentials.
    • Secure Credential Management: Deployment keys and credentials are managed within the workspace Settings.