Kedro-Viz Documentation

The data_ingestion pipeline is a modular component designed to transform raw data into structured, typed, and aggregated domain-level tables.

Workflow Steps:

1. Intermediate Level: Takes raw companies, shuttles, and reviews data and creates typed Parquet mirrors at the intermediate level.
2. Aggregation: Aggregates companies data to ensure a single row per company.
3. Primary Domain Level: Merges the three sources to create two specific primary tables:
   • prm_spine_table: Contains only relevant ID columns at the required grain. This table serves as the foundation for all subsequent feature and model_input tables, ensuring they maintain the same row count and column alignment.
   • prm_shuttle_company_reviews: Contains metrics intended for use as features or model inputs later in the pipeline.

The Kedro-Viz backend acts as the data provider and API layer for Kedro projects. Its primary responsibilities include:

• Data Access: The DataAccessManager interfaces with Repositories to fetch and structure data from the Kedro project.
• REST API: Exposes endpoints for pipeline structures and node-specific metadata to the frontend.
• CLI & Deployment: Provides a CLI for users to launch the visualization and supports building/deploying to static website hosting platforms.

Kedro-Viz uses a string data token to determine its data source. Depending on the token, it uses different loading strategies:

1. Bundled Data: Tokens like "test" or "demo" synchronously import mock JSON files from /src/utils/data/. Randomly generated data can be seeded using a seed query string in the URL to ensure reproducible layouts.
2. Asynchronous/External Data: In production or when using the "json" identifier, the app fetches data from API endpoints:
   • Pipeline Endpoints: Located at /api/pipeline/<id>. Loading a pipeline resets the pipeline state in the Redux store. The app defaults to /api/main (the 'default' pipeline) on first load.
   • Node Endpoints: Located at /api/nodes/<id>. These are called asynchronously when a user selects a node to populate the metadata panel.

In Kedro-Viz, a 'node' is a visual graph element used for rendering the flowchart. It is distinct from a standard Kedro node and can represent one of three types:

• task: A Kedro node (a Python function wrapper).
• data: A dataset.
• parameter: A reusable configuration variable.

An edge represents the link (input/output) between two of these nodes.

The modelling pipeline in this demo project manages the core machine learning lifecycle, specifically the split, train, and test stages.

Key architectural patterns used:

• Parametrized Data Splitting: The split_data method is designed to take parameters to generate a single source for both train and test datasets.
• Modular Pipeline Pattern: The pipeline uses Kedro's modular pattern to instantiate multiple versions of training and evaluation pipelines. In this example, it instantiates two separate pipelines using different Scikit-learn regressors: Random Forest and Linear Regression.

When you use the --save-file <path> option with kedro viz run, or when you use kedro viz build, Kedro Viz generates a specific directory structure. This structure can be reloaded later using the --load-file <path> option in the kedro viz run command.

The structure consists of:

• api/main: The main file containing the pipeline structure.
• api/nodes/: Contains JSON files for individual nodes (e.g., node1, node2).
• api/pipelines/: Contains JSON files for individual pipelines (e.g., pipeline1, pipeline2).

api/
├── main       # Main file containing pipeline structure
├── nodes/
│   ├── node1           # JSON files for individual nodes
│   ├── node2
│   └── ...
├── pipelines/
│   ├── pipeline1      # JSON files for individual pipelines
│   ├── pipeline2
│   └── ...

When hosting Kedro-Viz on AWS, keep the following in mind:

• Permissions: Kedro-Viz does not manage access control. You must use AWS IAM policies or S3 Access Control Lists (ACLs) to control who can view your visualization (e.g., restricting access to specific IPs or users).
• Billing: Kedro-Viz does not handle billing. You are responsible for AWS costs associated with S3 storage, which depend on object size, storage duration, and the chosen storage class.

Typography

Wrap all text elements in a .kedro class. This class establishes a base font-size of 10px, allowing all other font sizes to be set relatively using em (where 1em = 10px). This ensures consistent typography regardless of the parent application's styles.

Units

• Use em for styles dependent on font-size (e.g., text margins).
• Use % for relative layout positioning.
• Use px for absolute layout positioning.
• Do not use rem or viewport units (vh/vw/vmin/vmax) in code intended for the package library, as these can cause side-effects in the host application.

Kedro-Viz supports three distinct types of node previews:

1. Mermaid preview: Renders Mermaid.js visualizations such as flowcharts, sequence diagrams, and process flows. This is ideal for visualizing internal decision logic or workflows within a node.
2. Text preview: Displays text summaries, logs, usage examples, or testing information. It supports code snippets and syntax highlighting.
3. Image preview: Displays static diagrams, charts, or reference images using either URLs or base64-encoded data URIs.

Kedro-Viz uses the browser's window.localStorage API to persist user preferences across sessions. This includes settings such as:

• Node, tag, layer, sidebar, and label visibility
• UI flags
• Selected theme
• The active pipeline

The localStorage state is automatically updated via a Redux store subscriber whenever the store changes.