DOCSEARCHbyAlgolia
Submit Docs

Kedro Documentation

repository·main·Indexed 27 days ago

https://github.com/kedro-org/kedro

An open-source Python framework for creating production-ready, reproducible, and maintainable data engineering and data science pipelines. Kedro provides a project template, a data catalog for saving/loading data across various formats, pipeline abstraction with automatic dependency resolution, and support for orchestrators like Argo, Prefect, Kubeflow, AWS Batch, and Databricks.

Tokens
128.1K
Snippets
342
Records
666
Agent score
94%

What's inside Kedro

  1. Overview of Kedro CLI submodules

    main

    The kedro.framework.cli module provides the command-line interface for managing Kedro projects. It is organized into several submodules, each handling specific aspects of the Kedro lifecycle:

    • kedro (via cli): The main entry point for managing Kedro projects.
    • catalog: Commands for interacting with the Kedro Data Catalog.
    • pipeline: Commands for working with Kedro pipelines.
    • project: Commands for managing Kedro project structures and configurations.
    • registry: Commands for managing registered Kedro pipelines.
    • starters: Commands for creating new Kedro projects using templates.
    • server: Commands for running Kedro as an HTTP server.
    • hooks: Primitives to extend the behavior of the Kedro CLI using hooks.
    • jupyter: Helper functions for integrating Kedro with Jupyter and IPython environments.

  2. Overview of Kedro features

    main

    Kedro is a toolbox for production-ready data engineering and data science pipelines. Key features include:

    • Project Template: A standard, modifiable template based on Cookiecutter Data Science.
    • Data Catalog: Lightweight connectors for saving/loading data across various formats and file systems (local, cloud object stores, HDFS, etc.), including versioning support.
    • Pipeline Abstraction: Automatic dependency resolution between pure Python functions and visualization via Kedro-Viz.
    • Coding Standards: Support for pytest, Sphinx, ruff, and standard Python logging.
    • Flexible Deployment: Support for single/distributed machines and orchestrators like Argo, Prefect, Kubeflow, AWS Batch, and Databricks.

  4. Understand the mandatory Kedro project structure

    main

    For a project to be recognized as a Kedro project and support the kedro run command, it must contain three essential files:

    1. pyproject.toml: Defines the Python project and includes the [tool.kedro] section for project metadata.
    2. settings.py: Defines project-wide settings (logging, environment variables, paths) and allows for hooks registration and plugin configuration.
    3. pipeline_registry.py: A centralized place to register and access pipelines. It must contain a register_pipelines() function.

    While Kedro recommends a specific directory structure (including conf/, data/, src/, etc.), these three files are the minimum requirements for functionality.

  5. Understand Amazon EMR Serverless deployment limitations

    main

    When using Kedro with Amazon EMR Serverless, be aware of the following constraints:

    • Spark-Centric: EMR Serverless is designed for Spark jobs. It is not a general-purpose Python batch platform for non-Spark Kedro pipelines.
    • No Orchestration: EMR Serverless does not orchestrate the dependency order of multiple pipelines. You must manage job submission order manually or use tools like AWS Step Functions or Airflow.
    • Image-Based Deployment: Every deployment requires a custom container image. Any changes to code or dependencies require a full cycle: rebuild image $\rightarrow$ push to ECR $\rightarrow$ restart EMR application.
    • Resource Quotas: Jobs are subject to EMR Serverless service quotas and Spark limits regarding memory, vCPU, and runtime.
    • Compatibility: While mixed Spark and pandas jobs are supported, you must ensure Python and dependency versions are compatible with your specific EMR release (especially for legacy EMR 6.x).

  6. Deploy Kedro projects to Docker or Airflow

    main

    Kedro supports deployment to various targets through specialized plugins:

    • Docker: Use the kedro-docker plugin to package and ship projects within Docker containers.
    • Airflow: Use the kedro-airflow plugin to convert your Kedro project into an Airflow project.
    • Other targets: Support exists for AWS Batch, Prefect, and other third-party platforms via community plugins.

  7. Understand the execution flow of `kedro run`

    main

    When you execute the kedro run command, Kedro follows a specific lifecycle involving the creation of a KedroSession, loading the KedroContext, and executing a Runner. The process follows these high-level steps:

    1. Initialization: The KedroSession is created and the hook manager is initialized.
    2. Context Loading: The KedroContext is loaded, which includes the data catalog (supporting version loading/saving).
    3. Pipeline Selection: The selected pipeline is retrieved and filtered based on any command-line arguments provided.
    4. Execution: A Runner is created to execute the filtered pipeline.
    5. Hooks Lifecycle: During execution, the following hooks are triggered:
      • before_pipeline_run: Executed before the runner starts.
      • on_pipeline_error: Executed if the runner fails.
      • after_pipeline_run: Executed after the pipeline completes successfully.

  8. Understand the Kedro Data Catalog

    main

    The Data Catalog is a registry of all data sources available in a Kedro project. It is defined in a YAML file (typically catalog.yml) that maps dataset identifiers (used as names for node inputs and outputs) to specific dataset implementations and their configurations.

    Important Versioning Notes:

    • Since Kedro 0.19.0, datasets are no longer part of the core Kedro package. You must install and import them from the kedro-datasets package.
    • From kedro-datasets version 2.0.0 onwards, dataset names use a lowercase 's' (e.g., CSVDataset instead of CSVDataSet).

  9. Understand the Kedro CLI execution flow

    main

    When you run the kedro command, the execution follows a specific sequence to bootstrap your environment and load commands from various sources:

    1. Entrypoint: The command is triggered via the kedro.framework.cli:main entrypoint defined in pyproject.toml.
    2. Plugin Loading (Initialization): Kedro loads and runs all installed plugins registered under [project.entry-points."kedro.init"].
    3. Command Collection:
      • Built-in Commands: Collects global Kedro CLI commands (e.g., info, new, docs, starter).
      • Global Plugins: Loads commands from plugins registered under [project.entry-points."kedro.global_commands"].
    4. Project Bootstrapping: Kedro checks the current directory for a pyproject.toml to identify if it is within a Kedro project and bootstraps it.
    5. Context Enrichment: Project metadata is added to the click CLI context.
    6. Project-Specific Commands:
      • Project Plugins: Loads commands from plugins registered under [project.entry-points."kedro.project_commands"].
      • Project CLI: Loads commands defined within the local Kedro project (e.g., in cli.py).
    7. Execution: All collected command sets are combined and executed using the click library.

  10. Understand the Kedro security model

    main

    Kedro is a code authoring framework, not a multi-tenant web application. It does not provide isolation or restriction for user-written pipeline code. A Kedro project runs as standard Python in its deployment environment with the permissions provided by that environment.

    Core Security Principle: Kedro distinguishes between Code (logic authored by the developer, such as nodes.py, custom datasets, hooks, and plugins) and Data (inputs meant to configure or feed code, such as catalog.yml, parameters.yml, logging.yml, and dataset contents).

    The Golden Rule:

    • Code = responsibility (The developer is responsible for the safety of authored code).
    • Data becoming code = vulnerability (If Kedro's framework allows data to trigger unintended code execution, it is a vulnerability).

  11. Supported data locations and custom datasets

    main

    Kedro supports a wide range of datasets and data locations:

    • Data Locations: Kedro uses fsspec to support various data stores, including local file systems, network file systems, HDFS, and cloud object stores.
    • Custom Datasets: You can extend Kedro by adding support for proprietary data formats or filesystems by implementing custom dataset classes.

  12. Understand the Data Science pipeline structure

    main

    This modular pipeline demonstrates a standard machine learning workflow consisting of three nodes:

    1. train_model: Trains a simple multi-class logistic regression model.
    2. predict: Generates predictions using the trained model and a test dataset.
    3. report_accuracy: Calculates and reports the model's accuracy on the test set.