DOCSEARCHbyAlgolia
Submit Docs

dlt (data load tool)

repository·devel·Indexed 26 days ago

https://github.com/dlt-hub/dlt

An open-source python-first scalable data loading library designed to simplify extracting data from various sources, normalizing it, and loading it into over 20 supported destinations. It features a Dataset API for reading data into DataFrames and allows developers to declare data APIs using decorators. The library operates without requiring a backend to run.

Tokens
263.8K
Snippets
833
Records
1.3K
Agent score
90%

What's inside dlt

  1. Overview of dltHub capabilities

    devel

    dltHub is an agent-native data engineering platform built around the open-source dlt library. It provides an end-to-end workflow for:

    • Ingestion Pipeline Development: Build extract-and-load pipelines from REST APIs, SQL databases, cloud storage, and Python structures with schema inference and normalization.
    • Transformation Pipeline Development: Write Python-decorated transformations using @dlt.hub.transformation (in public preview) or integrate with dbt.
    • Pipeline Operations: Deploy, schedule, and monitor pipelines via the dltHub platform with support for cron and event-driven triggers.
    • Data Quality & Governance: Implement declarative correctness rules and tests to catch data issues.
    • Data Discovery & Serving: Access loaded data via typed Python/SQL Datasets, Marimo notebooks, or shareable public links.

  2. Overview of LLM-friendly documentation (llms.txt)

    devel

    The build process generates LLM-optimized output following the llms.txt specification via a custom Docusaurus plugin.

    Generated Artifacts:

    • /docs/llms.txt: An index of all master-version doc pages, grouped by sidebar category. This is the primary entry point for LLM agents.
    • /docs/hub/llms.txt: A separate index specifically for dltHub pages.
    • Markdown files: For every HTML page (e.g., /docs/general-usage/schema), a clean .md version is generated (e.g., /docs/general-usage/schema.md) with UI-only React components and MDX imports stripped out.

  3. Use the filesystem source to load files

    devel

    The filesystem source allows loading data from AWS S3, Google Cloud Storage, Google Drive, Azure Blob Storage, SFTP, local filesystems, and Public CDNs. It natively supports CSV, Parquet, and JSONL formats.

    Loading happens in two steps:

    1. Accessing files: The source identifies files via bucket_url and file_glob. You can filter by metadata or use incremental loading to only pick up new files.
    2. Reading content: A reader (like read_parquet) processes the file content and yields records.

  4. Understand load packages and the _dlt_loads table

    devel

    Each pipeline execution generates one or more load packages, uniquely identified by a load_id. This ID is added to data tables in a _dlt_load_id column and tracked in a special _dlt_loads table.

    Key concepts:

    • _dlt_loads table: Tracks complete loads with a status of 0. It includes schema_name, inserted_at, and schema_version_hash.
    • Data Integrity: In destinations that don't support long-running transactions (like Amazon Redshift), you can identify partially loaded data by filtering for load_ids that do not exist in the _dlt_loads table.
    • Transformations: You can chain transformations by updating the status column in _dlt_loads (e.g., starting at 0, updating to 1 after the first transformation, and so on).
    • Lineage: Use the pipeline_name and load_id to identify the source and time of data for troubleshooting or data vault architectures.

  5. Understand the dlt documentation structure

    devel

    The dlt documentation website is organized into two primary sidebars defined in sidebars.js:

    1. docsSidebar: The main sidebar for open-source dlt documentation. It includes manually curated sections like "Core concepts" and dynamically injected sections:

      • Code examples: Automatically appends all .md or .mdx files found under docs_processed/examples/.
      • API reference: Spliced into the "Reference" category if docs_processed/api_reference/sidebar.json (generated by pydoc) is present.

    2. hubSidebar: A secondary sidebar for dltHub-specific pages located under docs/hub/. It uses { type: 'ref', id: '...' } items to cross-reference open-source documentation, allowing seamless navigation between the two sidebars.

  6. Understand dlt Schema structure

    devel

    A dlt schema consists of tables and columns. You can access the dictionary of tables via the tables key or the tables property of a Schema object.

    Table Schema Properties

    • name and description
    • columns: A dictionary of column schemas.
    • write_disposition: A hint for how new data is loaded into the table.
    • schema_contract: Describes a contract on the table.
    • parent: A reference to a parent table (used in nested structures).

    Column Schema Properties

    • name and description
    • data_type: The column's data type.
    • precision: Precision for text, timestamp, time, bigint, binary, and decimal.
    • scale: Scale for the decimal type.
    • timezone: Indicates if the timestamp or time is TZ aware (default is true).
    • nullable: Boolean indicating if the column allows null values.
    • is_variant: Indicates if the column was generated as a variant of another column.

  7. Use the SQL Database verified source

    devel

    The sql_database verified source allows you to load data from SQL databases to your specified destination. It supports multiple backends for data retrieval, including SQLAlchemy, PyArrow, pandas, or ConnectorX.

    You can use two primary resources within this source:

    • sql_database: Reflects all tables and views in an SQL database and retrieves the data.
    • sql_table: Retrieves data from a specific SQL database table.

  8. Understand dltHub Datasets

    devel
    A dataset is a physical collection of data and dlt metadata at a destination. It includes the schema, load history, traces, and quality results. In dltHub, datasets act as the serving layer, allowing notebooks, dashboards, and downstream agents to read from a single source of truth containing schemas, annotations, and run metadata.

  9. Understand dlt core terminology

    devel

    To use dlt effectively, familiarize yourself with these core concepts:

    • Source: A location holding structured data (e.g., an API, spreadsheet, or database). In code, a source is the Python function/component that extracts data from that location.
    • Resource: A logical grouping of data within a source (e.g., an API endpoint, a spreadsheet tab, or a database table). In code, a resource is the Python function/component that extracts that specific data.
    • Destination: The target data store where data is loaded (e.g., Google BigQuery).
    • Pipeline: The process that moves data from a source to a destination by extracting, normalizing, and loading it based on the schema.
    • Schema: The definition of the normalized data structure (tables, column types, etc.) and the instructions for how dlt should process and load the data into the destination.
    • Config: Runtime values passed to a pipeline to control behavior (e.g., switching between local and production environments).
    • Credentials: A subset of configuration containing sensitive information that must be kept secret and not shared in plain text.

  10. Understand dltHub Platform Core Concepts

    devel

    The dltHub platform is a managed cloud environment for running dlt pipelines, transformations, and notebooks. It mirrors your local workspace into a workspace deployment in the cloud.

    Key Terminology

    • Job: A registered script in your workspace that defines code to run and an optional schedule.
    • Run: A single execution of a job with its own logs, status, and metadata.
    • Batch Jobs: Run with the prod profile; intended for scheduled data loading.
    • Interactive Jobs: Run with the access profile; intended for notebooks, dashboards, and Streamlit apps.
    • Profiles: Configuration sets that allow different environments (e.g., DuckDB for local development, MotherDuck for production, and read-only credentials for interactive sessions).

  11. Understand how dlt vault providers work

    devel

    Vault providers allow dlt to reconstruct a secrets.toml-like document from secrets stored in external vaults (like Google Cloud Secret Manager, AWS Secrets Manager, or Airflow Variables).

    Key behaviors:

    • Merging: The provider fetches TOML fragments and merges them into an in-memory document. More specific fragments (e.g., destination.bigquery) override more general ones (e.g., destination).
    • Caching: All lookups (including failed ones) are cached for the lifetime of the process. Changes in the vault require a process restart to take effect.
    • Lookup Order: The provider probes for configuration in this order:
      1. Global dlt_secrets_toml.
      2. Pipeline-scoped <pipeline_name>.dlt_secrets_toml.
      3. Known sections: sources, sources.<source_name>, destination, and destination.<destination_name>.
      4. Single-value lookups (if enabled).

  12. Understand Nested Table References

    devel

    When dlt normalizes nested data, it creates root and nested tables linked via nested references.

    Key components of a nested reference:

    • row_key: A unique identifier for a row (default name _dlt_id).
    • parent: A hint on the nested table specifying the parent table name.
    • parent_key: A column in the nested table referring to the parent's row_key (default name _dlt_parent_id).

    For merge write dispositions, dlt also uses a root_key (default name _dlt_root_id) to link nested tables directly back to the root table, skipping intermediate parents.