DOCSEARCHbyAlgolia
Submit Docs

kedro-mlflow

repository·master·Indexed 42 days ago

https://github.com/galileo-galilei/kedro-mlflow

A Kedro plugin that integrates MLflow for experiment tracking and pipeline deployment. It enables automatic logging of parameters, metrics, and datasets as artifacts, and provides an API to package entire Kedro pipelines as deployable MLflow models. Compatible with kedro>=0.16.0 and mlflow>=1.0.0; version 2.0.3 is specifically compatible with mlflow==3.x.

Tokens
27.4K
Snippets
81
Records
118
Agent score
30%

What's inside kedro-mlflow

  1. Overview of the kedro-mlflow plugin

    master

    kedro-mlflow is a Kedro plugin designed to integrate MLflow into Kedro projects. It provides a modular framework for high reproducibility of machine learning experiments and simplifies deployment.

    Key features include:

    • Automatic parameters tracking: Automatically logs parameters used in your pipelines.
    • Datasets tracking as artifacts: Tracks Kedro datasets as MLflow artifacts.
    • Pipeline as model: Packages Kedro pipelines as custom MLflow models for deployment and serving.
    • Automatic synchronization: Ensures synchronization between training and inference pipelines.

  2. What is kedro-mlflow?

    master

    kedro-mlflow is a Kedro plugin that provides lightweight and portable integration of MLflow capabilities into Kedro projects. It focuses on two core functionalities:

    1. Experiment Tracking: Enhances reproducibility by allowing you to effortlessly register parameters and datasets with minimal configuration during a Kedro run. These can be browsed in the MLflow UI and retrieved later.
    2. Pipeline as Model: Provides an API to convert a Kedro pipeline into a self-contained MLflow model. This model includes preprocessing, postprocessing, and training artifacts, allowing you to serve your Kedro pipeline as an API with minimal effort.

  3. Handle business objects in ML models

    master

    A robust machine learning model should handle raw business objects (e.g., a raw image, a customer profile, or a text string) rather than requiring the end-user to provide pre-processed tensors or encoded matrices.

    In kedro-mlflow, your model should be represented as a Kedro Pipeline that spans from the raw business object to the final prediction. The KedroPipelineModel class is designed to store and log these end-to-end pipelines, making them usable by downstream consumers without requiring them to reimplement your preprocessing logic.

  4. Understanding the role of kedro-mlflow in the ML lifecycle

    master

    Machine learning deployment is complex because it requires managing more than just code. While traditional software focuses on code, environment, and infrastructure, ML projects introduce two additional critical moving parts: parameters and data.

    kedro-mlflow is designed to address the challenges of this lifecycle by bridging the gap between iterative ML development and robust deployment. It helps solve several key issues:

    • Reproducibility: Managing the iterative nature of ML where experiments must be tracked to avoid losing track of the best models and to meet regulatory requirements.
    • Decoupling of Inference and Training: Ensuring that the model is not deployed in isolation, but as part of a pipeline that handles business objects. This prevents the risk of mismatch between preprocessing logic and model requirements.
    • Automation: Facilitating easy redeployment and retraining to combat model quality decay.

    By combining the pipeline management capabilities of Kedro (which handles the cyclic, stochastic development process) with the tracking capabilities of MLflow (which manages parameters and metrics), kedro-mlflow provides a framework for managing the full machine learning development lifecycle.

  5. Synchronize training and inference using PipelineML

    master

    To prevent desynchronization between training and inference, kedro-mlflow provides the PipelineML class. A PipelineML object binds a training Pipeline to an inference Pipeline via a shared input dataset name. This ensures that the inference logic is always packaged alongside the training logic.

    Use the pipeline_ml_factory function to create these objects. It is recommended to implement this within your Kedro hooks.py file using the register_pipelines hook.

    # hooks.py
from kedro_mlflow.mlflow import pipeline_ml_factory

class ProjectHooks:
    @hook_impl
    def register_pipelines(self) -> [str, Pipeline]:
        ml_pipeline = create_ml_pipeline()

        # convert your two pipelines to a PipelineML object
        training_pipeline_ml = pipeline_ml_factory(
            training=ml_pipeline.only_nodes_with_tags("training"),
            inference=ml_pipeline.only_nodes_with_tags("inference"),
            input_name="instances",
        )

        return {"__default__": training_pipeline_ml}

  6. Track parameters in MLflow

    master

    The plugin automatically records parameters used in your pipeline. There are two primary ways parameters are captured:

    1. Specific Parameters: Parameters accessed in your pipeline using the params: prefix (e.g., params:example_test_data_ratio).
    2. Global Parameters Dictionary: Passing the entire parameters dictionary from parameters.yml.

    Warning: Passing the entire parameters dictionary is considered bad practice. It makes it difficult to identify which specific parameters are actually used by a function and can lead to excessively long parameter names that may cause MLflow errors.

  8. What are MLflow Models?

    master

    MLflow Models are a standardized, agnostic format for storing machine learning models to ensure portability across different deployment environments. An MLflow Model typically consists of:

    • An MLModel file: A configuration file that tells MLflow how to load the model. It may include the model's Signature (input/output schema, including column names and order) and example data.
    • A conda.yml file: Specifies the virtual conda environment and package versions required to execute the model.
    • A model file: Usually model.pkl or python_function.pkl (for custom models) containing the trained model object.
    • An artifacts folder: Contains any additional data necessary for model execution.

    In the context of kedro-mlflow, you can convert entire Kedro Pipelines into MLflow models using pipeline_ml_factory and KedroPipelineModel because the Pipeline and DataCatalog objects provide the necessary metadata.

  11. Differentiate between an 'app' and a Kedro pipeline

    master

    In this framework, an "app" is a high-level organizational unit, whereas a "pipeline" is a technical Kedro construct.

    Differences include:

    • Composition: An app typically contains one or more Kedro pipelines.
    • Responsibility: Different teams or individuals often manage different apps.
    • Lifecycle: Apps have independent development and deployment lifecycles. For example, fixing a bug in the user_app should not trigger a resource-intensive retraining in the ml_app.
    • Orchestration: Apps have independent orchestration timelines and data storage requirements.
    • Communication: Apps communicate only through clearly defined data schemas (inputs/outputs).

  12. Enable automatic inference logging with MlflowHook

    master
    Instead of manually logging models, you can use MlflowHook.after_pipeline_run. This hook automatically detects if a run was a PipelineML object. If so, it automatically creates a KedroPipelineModel and logs it to MLflow. This ensures that the exact inference pipeline (code and artifacts) used during training is always synchronized and available in the MLflow UI.