DOCSEARCHbyAlgolia
Submit Docs

MatrixOne Documentation

repository·main·Indexed 21 days ago

https://github.com/matrixorigin/matrixone

A MySQL-compatible, cloud-native HTAP database integrating transactional, analytical, full-text, and vector search capabilities. Features 'Git for Data' version control (snapshots, branching, time travel), a CGO kernel with GPU-accelerated vector indexing via NVIDIA cuVS and RAFT, and a Python SDK for database interaction and ORM support.

Tokens
289K
Snippets
559
Records
1.1K
Agent score
81%

What's inside MatrixOne

  1. What is MatrixOne?

    main

    MatrixOne is an HTAP (Hybrid Transactional/Analytical Processing) database that features a unified HSTAP engine. It is designed to handle OLTP, OLAP, full-text search, and vector search within a single system without the need for ETL or data migration.

    Key characteristics include:

    • Git-style Version Control for Data: Provides features like instant snapshots (zero-copy), time travel (querying historical states), branching/merging for testing, instant rollbacks, and full audit trails.
    • MySQL Compatibility: Acts as a direct replacement for MySQL 8.0, allowing existing tools, ORMs, and applications to work without code changes.
    • AI-Native: Built-in support for vector retrieval (IVF/HNSW) and full-text search, making it suitable for RAG (Retrieval-Augmented Generation) applications.
    • Cloud-Native: Features separation of storage and compute, Kubernetes-native deployment, and elastic scaling.

  2. Overview of MatrixOne Vector Search capabilities

    main

    MatrixOne provides a comprehensive suite of vector operations designed for AI and machine learning applications. Key capabilities include:

    • Modern Vector API: High-level operations managed via vector_ops and vector_query.
    • Multiple Index Types: Support for both HNSW and IVF indexes to balance performance and accuracy.
    • Index Health Monitoring: Use get_ivf_stats() to monitor IVF index balance and performance (critical for production).
    • Flexible Data Types: Support for vecf32 and vecf64 vector types with configurable dimensions.
    • Distance Metrics: Support for L2, cosine, inner product, and negative inner product calculations.
    • ORM Support: Full integration with SQLAlchemy, allowing vector types to be used within standard ORM models.
    • Performance Optimization: Built-in query optimization and index management.

  3. Overview of MatrixOne Fulltext Search capabilities

    main

    MatrixOne's Python SDK provides advanced fulltext search capabilities designed for text analysis and information retrieval. Key features include:

    • Multiple Algorithms: Support for both TF-IDF and BM25 algorithms.
    • Search Modes: Support for Natural language and Boolean search modes.
    • Client Interface: Dedicated client methods for performing fulltext operations.
    • ORM Integration: Seamless integration with SQLAlchemy models for managing text data.
    • Multi-language Support: Support for various languages, including Chinese.
    • JSON Search: Ability to perform fulltext searches within JSON documents.

  4. MatrixOne Python SDK API Reference Overview

    main

    The MatrixOne Python SDK provides a comprehensive suite of tools for interacting with the MatrixOne database. The API is organized into several functional domains:

    • Core Connectivity: Synchronous (client) and Asynchronous (async_client) clients for database connection.
    • ORM and Query Building: Object-Relational Mapping (ORM) classes and query builders for high-level data interaction.
    • Vector Operations: Specialized tools for handling vector_type, vector_index, and vector_manager for AI-driven workloads.
    • Fulltext Search: Capabilities for managing fulltext_index and performing fulltext_search.
    • Data Management: Tools for data lifecycle including stage_manager, load_data_manager, export_manager, and metadata_manager.
    • Database Management: Administrative operations such as snapshot_manager, account_manager, restore_manager, clone_manager, pitr_manager (Point-in-Time Recovery), pubsub_manager, moctl_manager, branch_builder, and clone_builder.
    • Utilities: Helper tools including result_set handling, transaction_wrapper, version_manager, logger, and cli_tools.

  5. MatrixOne Python SDK Features Overview

    main

    The MatrixOne Python SDK provides a comprehensive suite of tools for interacting with the MatrixOne database, including:

    • Database Operations: SQLAlchemy-like interface, async support via AsyncClient, and connection pooling.
    • Advanced Search: Vector similarity search (HNSW/IVF with L2, Cosine, Inner Product) and Fulltext search (BM25/TF-IDF).
    • Data Management: Pandas-style bulk loading (read_csv, read_json, read_parquet) and exporting (to_csv, to_jsonl) with support for local files and external stages (stage://).
    • Lifecycle & Versioning: Snapshot management, Point-in-Time Recovery (PITR), Table Cloning, and Git-style Branch Management.
    • Maintenance: Metadata analysis, secondary index verification, and the mo-diag CLI tool for diagnostics.

  6. MatrixOne Python SDK Example Categories

    main

    The SDK examples are organized into several functional categories to help you explore different features:

    • Basic Operations: Connection, account management, async operations, and transaction management.
    • Advanced Features: Snapshot/restore, SQLAlchemy integration, Pub/Sub messaging, and logging.
    • Vector Search: Vector basics, index creation, similarity search, and advanced operations.
    • Fulltext Search: JSON Parser and NGRAM Parser examples.
    • ORM (Object-Relational Mapping): Snapshot ORM, SQLAlchemy-style ORM, engine integration, and advanced ORM features.
    • Git-Style Branch Operations: Complete branch operations including ORM support, development workflows, safe data migration, A/B testing, and multi-team development.
    • Other: SQL builder, CDC (Change Data Capture) task lifecycle, metadata operations, and query/update operations.

  7. Features of the MatrixOne Python SDK

    main

    The MatrixOne Python SDK provides a high-level interface for several advanced database operations:

    • Core Operations: High-performance connection pooling and full async/await support via AsyncClient.
    • Search:
      • Vector Search: HNSW and IVF indexing (supporting f32/f64) with distance metrics (L2, Cosine, Inner Product) and IVF health monitoring via get_ivf_stats().
      • Fulltext Search: BM25 and TF-IDF indexing with natural language and boolean modes.
    • Data Management:
      • Export: Pandas-style to_csv() and to_jsonl() methods supporting local files or stage:// protocol.
      • CDC: Change Data Capture management via client.cdc or async_client.cdc.
      • Snapshots & Recovery: Snapshot management, Point-in-Time Recovery (PITR), and Table Cloning.
    • Administration: Account/role management, Pub/Sub support, and metadata analysis.
    • Integration: Seamless SQLAlchemy ORM integration and full type safety.

  8. Overview of MatrixOne Pub/Sub

    main

    MatrixOne's Pub/Sub system enables real-time messaging and event-driven architectures. Key features include:

    • Real-time Messaging: Publish and subscribe to topics.
    • Event-driven Architecture: Build reactive applications.
    • Topic Management: Create, list, and manage topics.
    • Message Filtering: Subscribe to specific message patterns.
    • Async Support: Full async/await support for high-performance applications.
    • Durability: Reliable message delivery with persistence options.
    • Transaction Support: Atomic pub/sub operations using session().

  10. What is MatrixOne CDC?

    main

    Change Data Capture (CDC) is a real-time data replication feature in MatrixOne. It captures and synchronizes data changes (INSERT, UPDATE, and DELETE operations) from a source MatrixOne database to a target MySQL-compatible database.

    Key Capabilities

    • Multi-Level Replication: Supports replication at the Account, Database, or Table level.
    • Initial Snapshot: Can perform an optional full data snapshot before starting incremental synchronization.
    • Bounded Replication: Supports time range control using start and end timestamps.
    • Filtering: Allows including or excluding specific databases or tables.
    • Resilience: Features automatic recovery for transient errors and state management to pause, resume, or restart tasks without data loss.

    Supported Targets

    • MySQL 5.7+
    • MySQL 8.0+
    • MatrixOne (for inter-cluster replication)