Apache Paimon Documentation

Paimon Virtual Storage (PVFS) provides a way to access tables within a Paimon REST Catalog using human-readable virtual paths instead of the internal UUID-based paths generated by the catalog.

Key benefits:

• Simplified Access: Use paths like pvfs://catalog_name/database_name/table_name/ to access Paimon Tables, Format Tables, and Object Tables.
• Unified Permissions: Access is managed through the Paimon REST Catalog's permission system, eliminating the need for a separate file system permission layer.
• Transparent Mapping: While users interact with virtual paths, the system automatically maps these to the true underlying file system paths for reading and writing.

Paimon supports real-time data ingestion from various sources with schema evolution. Added columns are synchronized to Paimon tables in real time without requiring a job restart. Supported synchronization methods include:

• MySQL: Synchronize a single table or an entire database.
• Kafka: Synchronize a single topic or a database (multiple topics/tables).
• MongoDB: Synchronize a single collection or an entire database.
• Pulsar: Synchronize a single topic or a database (multiple topics/tables).
• Program API: Use the Flink API to synchronize custom DataStream inputs into a Paimon table.

Apache Paimon is a lake format designed for building Realtime Lakehouse Architectures. It enables both streaming and batch operations using Flink and Spark. Paimon combines lake format with an LSM (Log-Structured Merge-tree) structure to bring realtime streaming updates into the lake architecture.

Formerly known as Flink Table Store, its architecture incorporates design concepts from Apache Iceberg.

Apache Paimon is a Data Lake Platform designed for unified batch, streaming, and multimodal AI workloads using a single lake format. It provides three primary functional pillars:

1. Large-Scale Analytics

Supports petabyte-scale tables with features including:

• Time travel
• Fast scan planning
• Schema evolution
• Incremental clustering
• Append-only tables and streaming appends

2. Realtime Streaming

Utilizes LSM-powered streaming updates with:

• Multiple merge engines
• Changelog producers
• PrimaryKey tables for compaction and streaming updates
• CDC Ingestion pipelines (MySQL, PostgreSQL, Kafka, MongoDB, Pulsar)

3. Multimodal AI

Supports AI and machine learning workloads via:

• Vector search and full-text search
• Blob tables and blob storage
• Global indexes
• PyPaimon: A native Python SDK with integration for Ray, PyTorch, and Pandas.

Multimodal Table extends the standard Append Table to support storing and querying multimodal data such as images, videos, audio, vectors, and full-text content within a single table. It leverages the Data Evolution mode to allow efficient partial column updates and schema evolution without rewriting entire data files.

Key features include:

• Data Evolution: Efficient schema evolution and partial column updates.
• Variant Storage: Support for schema-flexible semi-structured data (can be used in regular Paimon tables).
• Blob Storage: Storage of large binary objects in dedicated .blob files.
• Vector Storage: Management of vector embeddings in optimized Vortex-format files.
• Global Index: Support for BTree, Bitmap, vector, and full-text indexes for lookups and similarity searches.

Paimon Lumina integrates the Lumina vector search library as a vector index for Apache Paimon's global index framework. It uses JNI to access the Lumina native library via the lumina-jni artifact.

Architecture Restriction: The Lumina native library only supports x86_64 (AMD64) architecture. It is not compatible with ARM architectures (such as Apple Silicon or aarch64).

The BLOB (Binary Large Object) type is designed for storing multimodal data like images, videos, audio, and large binary objects. Unlike the BYTES type which stores data inline, BLOB stores large binary data in separate .blob files and maintains references, improving performance for large objects.

Supported types include:

• BLOB (scalar)
• ARRAY<BLOB> (ordered collection)
• MAP<K, BLOB> (keyed blobs)

Blob storage is based on Data Evolution mode and requires row-tracking.enabled and data-evolution.enabled to be set to true for correct functionality.

Supported Computation Engines

In addition to Apache Flink, Paimon can be read by:

• Apache Spark
• StarRocks
• Apache Doris
• Apache Hive
• Trino

Internal Storage Mechanism

• Data Format: Stores columnar files on a filesystem or object store.
• Metadata: Uses manifest files to manage metadata, enabling large-scale storage and data skipping.
• Primary Key Tables: Utilizes an LSM tree structure to support high-volume data updates and high-performance queries.

Paimon's metrics system measures behaviors such as reading and writing (e.g., manifest files scanned, commit duration, files deleted during compaction). Metrics are reported at the table granularity.

There are three supported metric types:

• Gauge: A value of any type at a specific point in time.
• Counter: A value that is incremented or decremented to count occurrences.
• Histogram: Measures the statistical distribution of a set of values, providing min, max, mean, standard deviation, and percentiles.