DOCSEARCHbyAlgolia
Submit Docs

dlt (data load tool)

website·Indexed 35 days ago

https://dlthub.com/docs/

A Python library for building production-grade data pipelines to extract, normalize, and load data from various sources into destinations. It provides verified sources for REST APIs, OpenAPI generators, and over 30 SQL databases, supporting incremental loading, schema management, and memory-efficient data extraction.

Tokens
260.6K
Snippets
1.3K
Records
1.7K
Agent score
72%

What's inside dlt

  1. Overview of dltHub capabilities

    dltHub is an agent-native platform that extends the open-source dlt library. It provides a toolchain designed for coding agents (like Claude Code, Codex, and Cursor) to build and operate the following end-to-end data workflows:

    • Ingestion: Build extract-and-load pipelines from REST APIs, SQL databases, and cloud storage with workspace scaffolding and AI-driven generation.
    • Transformations: Create Python-decorated transformations using @dlt.hub.transformation or integrate with dbt.
    • Pipeline Operations: Deploy, schedule (cron/event-driven), and monitor pipelines via the dltHub platform (app.dlthub.com) using isolated profiles for dev, prod, and access environments.
    • Data Quality & Governance: Implement declarative correctness rules and tests to catch data issues.
    • Data Discovery & Serving: Access loaded data via typed Python/SQL datasets and build shareable data apps using Marimo notebooks.

  2. Overview of Google Analytics verified source

    The dlt verified source for Google Analytics loads data from the Google Analytics API into a destination of your choice. It provides three main resources:

    • google_analytics: Loads basic Analytics information.
    • metrics_table: Assembles data relevant to the report's metrics.
    • dimensions_table: Compiles data related to the report's dimensions.

  3. Overview of dltHub Platform and Tools

    dltHub provides a suite of managed and source-available tools to extend the open-source dlt library for production environments. It includes:

    • dltHub Platform: A hosted Web UI and managed runtime for deploying, scheduling, and monitoring pipelines, managing workspaces, and collaborating.
    • AI Toolkits (AI Harness): A collection of skills, rules, and MCP wiring designed to drive agentic pipeline development within AI coding assistants like Claude Code, Cursor, and Codex.
    • dltHub Context: Source-specific agent contexts (specs, prompts, and documentation) used to prime AI assistants for building pipelines for thousands of APIs.
    • dlthub library: A Python package that adds production capabilities including data quality checks, Python transformations, dbt transformations, and premium sources/destinations (e.g., Iceberg, DuckLake, and MSSQL Change Tracking).

  4. Overview of the dlt filesystem source

    The filesystem source allows loading structured files (CSV, Parquet, and JSONL) from various locations into a data pipeline. Supported locations include:

    • AWS S3
    • Google Cloud Storage (GCS) and Google Drive
    • Azure Blob Storage
    • Remote filesystems via SFTP
    • Local filesystem
    • Public CDNs

    Loading happens in two steps: first, accessing and filtering files by metadata or name (with support for incremental loading), and second, reading the file content via a reader to yield records.

  5. Overview of Facebook Ads verified source

    The facebook_ads verified source for dlt loads data from the Facebook Marketing API into a destination of your choice. It supports the following endpoints:

    • campaigns: Structured marketing initiatives focusing on specific goals.
    • ad_sets: Subsets or groups of ads within a campaign.
    • ads: Individual advertisements within an ad set.
    • creatives: Visual and textual elements of an advertisement.
    • ad_leads: Information from users interacting with lead generation ads.
    • facebook_insights: Audience demographics, post reach, and engagement metrics.

  6. Overview of supported dlt destinations

    dlt provides a wide range of built-in destinations for loading data into various storage systems, databases, and vector stores. Supported destinations include:

    • Object Stores & Filesystems: AWS S3, Google Cloud Storage, Azure Blob Storage.
    • Cloud Data Warehouses: BigQuery, Snowflake, Amazon Redshift, Databricks, Azure Synapse, Microsoft Fabric Warehouse.
    • SQL Databases: Postgres, MS SQL, and any database supported via SQLAlchemy.
    • Analytical Databases & Lakehouses: DuckDB, MotherDuck, ClickHouse, Dremio, Delta Lake, Iceberg, Lance, LanceDB.
    • Vector Databases (AI/ML): Qdrant, Weaviate.
    • Specialized Destinations: Hugging Face Datasets, DuckLake.
    • Custom Implementations: Support for custom destination functions for Reverse ETL and community-contributed destinations.

  7. Overview of the dlt pipeline architecture

    dlt (data load tool) automatically transforms data from sources (APIs, databases, Python structures) into a live dataset in a destination (e.g., BigQuery, DuckDB, Delta Lake). It supports data formats including JSON objects, Python lists/dictionaries, pandas DataFrames, and Arrow tables. The core building block is the pipeline, which orchestrates data movement through three discrete phases: Extract, Normalize, and Load.

  8. Overview of dlt (Open Source) vs dltHub (Commercial)

    The dlt ecosystem consists of two primary offerings:

    1. dlt (OSS): A free, open-source (Apache 2.0) Python library used to build ingestion pipelines. It is a code-first tool that runs anywhere Python runs and is suitable for users who manage their own orchestration and scheduling.
    2. dltHub: A paid, commercial agentic platform that provides a managed runtime to deploy, schedule, and monitor dlt pipelines. It includes advanced features such as data quality metrics, transformation pipelines (dltHub/dbt), the AI Harness for coding agents, and premium sources/destinations (e.g., MS SQL, Iceberg, Delta, Snowflake + Iceberg/Open Catalog).

  9. Overview of the Google Ads verified source

    The google_ads verified source allows loading data from the Google Ads API into a destination of your choice. It supports the following resources:

    • customers: Businesses or individuals who pay to advertise their products.
    • campaigns: Structured sets of ad groups and advertisements.
    • change_events: Modifications made to an account's ads, campaigns, and related settings.
    • customer_clients: Accounts that are managed by a given account.

  10. Overview of dlt configuration and credentials management

    The dlt configuration mechanism allows developers to define credentials for external systems and pipeline settings separately from the source code. This ensures sensitive information is kept out of version control and allows for runtime behavior adjustments (such as performance optimization) without modifying code. Key capabilities include built-in support for common systems, auto-generated configuration specs for functions decorated with @dlt.source, @dlt.resource, and @dlt.destination, and comprehensive configurability across pipelines, normalizers, loaders, and logging.

  11. Overview of DuckLake destination

    DuckLake is a lakehouse-style destination built on the DuckDB engine using the ducklake extension. It stores dlt tables as files on a filesystem or object store while maintaining table metadata in a separate SQL catalog. To use DuckLake, you must provide a SQL catalog (e.g., SQLite, DuckDB, Postgres, MySQL) and a storage backend (e.g., local files, S3, GCS, ABFSS).

  12. Overview of Postgres replication verified source

    The pg_replication verified source enables Change Data Capture (CDC) from Postgres using logical decoding and the pgoutput output plugin. It provides two primary resources: init_replication for setup and initial snapshots, and replication_resource for streaming published messages from a replication slot. Note that this source does not currently support the scd2 merge strategy.