DOCSEARCHbyAlgolia
Submit Docs

Elyra Documentation

repository·main·Indexed 21 days ago

https://github.com/elyra-ai/elyra

Elyra is a collection of AI-centric extensions for JupyterLab providing visual pipeline editing, batch job execution for scripts and notebooks, and enhanced code editing for Python and R. It includes connectors for Apache Airflow and Airflow provider packages to load operator definitions from .whl distributions into the Visual Pipeline Editor. The documentation covers development and testing with Cypress, server-side extension configuration, and building custom Docker images for standalone, JupyterHub, and Kubeflow environments.

Tokens
51.5K
Snippets
91
Records
311
Agent score
83%

What's inside Elyra

  1. Overview of Elyra features

    main

    Elyra provides a suite of AI-centric extensions for JupyterLab designed to enhance the data science workflow. Key capabilities include:

    • Visual Pipeline Editor: For designing AI pipelines visually.
    • Batch Job Execution: Ability to run notebooks, Python, or R scripts as batch jobs.
    • Code Management: Reusable code snippets and integrated Python/R script editors with local/remote execution.
    • AI Assistance: Integration with jupyter-ai for AI-powered code assistance.
    • Advanced Navigation: Auto-generated Table of Contents for notebooks and Python scripts.
    • Runtime & Debugging: Hybrid runtime support via Jupyter Enterprise Gateway and an experimental integrated debugger for Python scripts.
    • Developer Tools: Language Server Protocol (LSP) integration and Git version control.

  2. Overview of Elyra AI-centric extensions

    main

    Elyra is a set of extensions for JupyterLab designed to enhance AI/ML workflows. Its core capabilities include:

    • AI Pipelines Visual Editor: A tool for building and modularizing AI pipelines from notebooks, Python scripts, and R scripts. Pipelines can be executed locally, on Kubeflow Pipelines, or with Apache Airflow.
    • Batch Job Execution: The ability to run notebooks, Python scripts, or R scripts as batch jobs in local or remote cloud environments using the Run as Pipeline button.
    • Reusable Code Snippets: An editor for managing and reusing code fragments to reduce repetitive tasks.
    • Hybrid Runtime Support: Integration with Jupyter Enterprise Gateway to allow notebooks to share resources across distributed clusters like Apache Spark, Kubernetes, or OpenShift, enabling seamless use of cloud resources like GPUs and TPUs.
    • Enhanced Script Editors: Specialized editors for Python and R with support for local/remote execution and integration with JupyterLab's debugger.
    • Enhanced Navigation: Auto-generated Table of Contents for both notebooks (via markdown headings) and Python scripts.
    • LSP Integration: Support for Language Server Protocol (LSP) providing IDE-like features such as autocompletion, code navigation, and linting.
    • Git Integration: Built-in version control for cloning, tracking, and managing changes within JupyterLab.

  3. Explore Elyra articles and blog posts

    main

    A collection of external resources, including Medium articles and blog posts, covering advanced Elyra topics such as:

    • Pipeline Configuration: Enabling the pipeline editor for specific runtimes and using pipeline parameters.
    • Customization: Customizing Elyra pipelines and using custom components.
    • Automation: Automating tasks using the Elyra CLI and integrating with Apache Airflow.
    • Integrations: Using Apache Airflow Operators and Kubeflow Pipelines with Elyra.
    • Feature Updates: Historical overviews of new features in Elyra versions (e.g., 2.0, 2.2, 3.3).

  4. Use the elyra-pipeline CLI to manage workflows

    main

    The elyra-pipeline command is used to interact with Elyra pipelines, which represent workflows that can run locally or remotely on Kubeflow Pipelines or Apache Airflow. Use this CLI to:

    • Run pipelines locally
    • Submit pipelines for remote execution
    • Validate pipelines
    • Describe pipelines (summary and dependencies)
    • Export pipelines
    elyra-pipeline

  5. Use the elyra-metadata CLI to manage metadata

    main
    The elyra-metadata command is used to manage Elyra metadata such as code snippets, runtime configurations, runtime images, and pipeline components. You can use it to list, create, update, export, import, or delete metadata instances. The CLI is part of the Elyra installation and can be used independently of a running JupyterLab instance.
    elyra-metadata

  6. What is a runtime configuration in Elyra

    main

    A runtime configuration provides Elyra with the necessary access and connectivity details to interact with external resources for scalable pipeline execution. These resources typically include:

    • Orchestration Engines: Kubeflow Pipelines (v2 supported) or Apache Airflow (2.7.0+ and 3.x supported; 1.x is no longer supported).
    • Object Storage: S3-compatible storage (e.g., Minio or IBM Cloud Object Storage) for storing pipeline artifacts.

    Runtime configurations can be managed via the JupyterLab UI or the elyra-metadata CLI.

  7. Metadata Service Components

    main

    The Metadata Service is composed of the following key components:

    • Schemaspace: A container for schemas with an id (UUID), name, display_name, and description.
    • SchemasProvider: A class (discovered via entrypoints) that provides schemas. Typically, one provider corresponds to one schemaspace.
    • Metadata: The internal class used to house JSON instance data. It can be subclassed to hook into lifecycle operations (load, create, update, delete) on a per-schema basis.
    • SchemaManager: A singleton that loads and houses all schemas organized by Schemaspace. It validates all schemas provided by SchemasProvider against a meta-schema.
    • MetadataManager: The primary interface for operating on Metadata instances. It handles validation (via SchemaManager) and drives persistence/retrieval via a MetadataStore.
    • MetadataStore: An abstract base class for persistence. The default implementation is FileMetadataStore.
    • REST/CLI API: Interfaces for manipulating metadata entities via /elyra/metadata, /elyra/schema, and /elyra/schemaspace endpoints, or the elyra-metadata CLI tool.

  8. Prerequisites for Runtime Image Configurations

    main

    A runtime image configuration identifies a container image used to run pipeline nodes on platforms like Kubeflow Pipelines or Apache Airflow. The container image must meet the following requirements:

    • Registry Access: The image must be stored in a container registry (e.g., hub.docker.com or a private intranet registry) accessible by the container platform.
    • Python: Must have a current Python 3 version pre-installed with python3 in the search path.
    • Curl: Must have curl pre-installed and in the search path.

  10. How Component Catalogs work in Elyra

    main

    Elyra does not host its own component repository. Instead, it uses Component Catalogs to pull components from external sources via a connector API. You can configure Elyra to fetch components from various locations, which are then exposed in the pipeline editor palette.

    Supported built-in catalog types include:

    • Filesystem: Accesses specific component files (e.g., .yaml) via a path.
    • Directory: Recursively searches a directory for component files.
    • URL: Retrieves components via HTTP GET requests (supports http, https, and file schemes).
    • Apache Airflow package/provider catalogs: (Deprecated in Elyra 4) Accesses operators from Airflow 1.x wheel distributions.

    You can extend Elyra by installing new connectors from the marketplace or implementing your own.

  12. Understanding Elyra pipelines and components

    main

    A pipeline is a collection of nodes connected to define execution dependencies. Each node is an instance of a component that performs a specific unit of work (e.g., loading data, training a model).

    There are two main types of pipelines:

    1. Generic pipelines: Comprised only of generic components (e.g., Jupyter notebooks, Python scripts, R scripts). These are portable and can run locally in JupyterLab, on Kubeflow Pipelines, or on Apache Airflow.
    2. Runtime-specific pipelines: Associated with a specific platform (Kubeflow Pipelines or Apache Airflow) and may include custom components specific to that runtime.

    Note: You cannot convert a generic pipeline to a runtime-specific one, or vice versa.