DOCSEARCHbyAlgolia
Submit Docs

OpenMLDB Documentation

repository·main·Indexed 23 days ago

https://github.com/4paradigm/openmldb

OpenMLDB is an open-source machine learning database and feature platform that enables 'Development as Deployment' using SQL to define features consistent across offline training and online inference. The documentation covers installation, Java demo project setup, and performance benchmarking for online SQL engines, memory usage, and query performance compared to Redis. It also provides detailed guides for predicting taxi trip durations in both Cluster and Standalone modes.

Tokens
168.3K
Snippets
421
Records
817
Agent score
79%

What's inside OpenMLDB

  1. Overview of Data Definition Language (DDL) in OpenMLDB

    main
    OpenMLDB uses Data Definition Language (DDL) statements to manage the structure of the database, including databases, tables, indexes, and user-defined functions (UDFs). You can use these statements to create, modify, and delete the schema components required for your machine learning feature platform.

  2. Overview of OpenMLDB Operations CLI

    main

    The Operations CLI is used to manage the underlying infrastructure and administrative tasks of OpenMLDB. It is divided into two primary client interfaces:

    1. NS Client (Namespace Client): Used for managing namespaces and high-level administrative operations.
    2. Tablet Client: Used for managing data tablets, replicas, and data migration tasks.

  3. Overview of FeatInsight Feature Store Service

    main

    FeatInsight is a feature store service built on top of OpenMLDB designed for efficient feature computation, management, and orchestration. It provides a unified interface for the entire machine learning feature engineering lifecycle, including:

    • Data Import & Management: Importing and managing datasets and online data sources.
    • Feature Management: Storing original and generated features.
    • Offline Scenarios: Generating training datasets from offline data via distributed computing and managing offline tasks.
    • Online Scenarios: Deploying feature services that provide hard real-time online feature extraction APIs.
    • SQL Playground: Executing OpenMLDB SQL statements for feature calculations in both online and offline modes.
    • Computed Features: Storing pre-computed features directly into OpenMLDB online tables for high-speed reads and writes.

  4. Overview of Airflow OpenMLDB Provider Operators

    main

    The Airflow OpenMLDB Provider allows Apache Airflow to connect to the OpenMLDB API Server. It provides several specialized operators for managing data and deployments, all of which are built upon the OpenMLDBSQLOperator which supports standard SQL commands.

    Available Operators:

    • OpenMLDBLoadDataOperator: For loading data into OpenMLDB.
    • OpenMLDBSelectIntoOperator: For selecting data into OpenMLDB.
    • OpenMLDBDeployOperator: For deploying models or configurations.
    • OpenMLDBSQLOperator: The base operator that supports all SQL operations.

    Note: This provider includes operators and a hook, but does not include sensors.

  5. What is OpenMLDB and its core mission

    main
    OpenMLDB is an open-source machine learning database designed as a feature platform. Its primary mission is to enable Development equals Deployment, ensuring that feature engineering logic used during offline training is identical to the logic used during online inference. This eliminates the need to refactor Python-based data science scripts into production-ready C++ or database code, reducing the risk of data leakage and engineering overhead.

  6. Use the Kafka Connect JDBC Connector

    main

    The kafka-connect-jdbc is a Kafka Connector designed for loading data to and from any JDBC-compatible database. It allows for data ingestion and extraction between Kafka and various relational databases via the JDBC interface.

    For detailed configuration, parameter definitions, and operational guidance, refer to the official Confluent documentation.

  7. Key features and highlights of OpenMLDB

    main

    OpenMLDB provides several enterprise-grade capabilities for ML applications:

    • Consistent Features: Guarantees identical feature logic for both offline training and online inference via the unified execution plan generator.
    • Ultra-Low Latency Real-Time Features: The real-time SQL engine is optimized for time-series data, achieving millisecond-level response times.
    • SQL-based Feature Definition: Uses SQL as the unified language, featuring extended syntax for feature engineering like LAST JOIN and WINDOW UNION.
    • Production-Ready Infrastructure: Supports distributed storage/computing, fault recovery, high availability, seamless scale-out, monitoring, and heterogeneous memory support.

  9. Integrate Apache Pulsar as an online data source for OpenMLDB

    main

    Apache Pulsar can serve as an online data source for OpenMLDB, enabling real-time data streams to be imported into OpenMLDB online. This is achieved using the OpenMLDB JDBC Connector for Pulsar.

    The integration process follows three main steps:

    1. OpenMLDB Setup: Create the target databases and tables in OpenMLDB.
    2. Pulsar Setup: Create a sink in Pulsar to connect the Pulsar data stream to OpenMLDB and configure the corresponding schema in Pulsar to ensure data is correctly received and stored.
    3. Testing: Verify the data flow from Pulsar to OpenMLDB.

  10. Overview of OpenMLDB Monitoring Architecture

    main

    OpenMLDB monitoring uses a combination of tools to collect and visualize metrics:

    • Prometheus: Used to collect monitoring metrics.
    • Grafana: Used to visualize metrics.
    • OpenMLDB Exporter: Exposes database-level metrics.
    • Component Metrics: Each OpenMLDB component (Nameserver, Tablet, etc.) exposes its own component-level metrics.
    • node_exporter: Used to expose machine and operating system related metrics.

  11. What is OpenMLDB and its core philosophy

    main

    OpenMLDB is an open-source machine learning database that serves as a production-grade feature platform. It is designed to provide consistent feature computing for both offline training and online inference, solving the problem of 'online-offline inconsistency' in AI engineering.

    Core Philosophy: "Develop Once, Deploy Anywhere"

    OpenMLDB aims to minimize the gap between data scientists (who develop features) and engineering teams (who deploy them). The workflow is optimized into three steps:

    1. Offline Development: Use SQL to develop feature scripts for model training.
    2. One-click Deployment: Deploy SQL feature scripts directly, switching from offline mode to online mode.
    3. Online Inference: Connect to real-time data for millisecond-latency feature computation used in model inference.

    Key Architectural Components

    • Unified SQL: A single programming language for both batch and real-time processing.
    • Real-time SQL Engine: A high-performance engine providing millisecond-level latency.
    • Batch SQL Engine: Based on the OpenMLDB Spark distribution for large-scale offline processing.
    • Consistency Execution Plan Generator: The component that links real-time and batch engines to ensure identical computation logic.

  12. Prevent duplicate data with PutIfAbsent

    main

    The PutIfAbsent option acts like INSERT OR IGNORE, preventing the insertion of duplicate data. This is useful for retrying failed import jobs.

    Performance Considerations

    • Supported: Only available in cluster mode (not supported in local mode).
    • Efficient: When the table has a ts index and the volume of data for the same key+ts is < 10k rows, performance impact is minimal (approx. 2x regular import time).
    • Inefficient: If the table has a time index (null ts column) or the volume for the same key+ts is > 100k rows, performance can degrade significantly (over 10x regular import time). In these cases, deduplicate data before importing.