Apache Amoro Documentation

The amoro-web project follows this structure:

• mock/: Mock data files.
• public/: Static assets like index.html and favicon.ico.
• src/: Main source code.
  • src/components/: Shared components used across multiple views.
  • src/views/: Page-level views.
  • src/services/: Services used for communication with the AMS server.
  • src/hooks/: Custom Vue hooks.
  • src/store/: State management.
  • src/utils/: Utility functions.
  • src/main.tsx: The project entry point.
  • src/App.vue: The root Vue component.
• vite.config.ts: Vue 3 configuration file.
• package.json: Project build scripts and dependency definitions.
• pnpm-lock.yaml: Dependency lock file.

Amoro supports four primary table formats depending on your use case:

• Iceberg format: Use this to entrust existing Iceberg tables to Amoro for maintenance. You retain all native Iceberg functions while gaining Amoro's performance and stability enhancements.
• Mixed-Iceberg format: Optimized for streaming update scenarios. Choose this if you have high-performance requirements for streaming updates or need CDC (Change Data Capture) incremental data reading functions.
• Mixed-Hive format: Designed for users with existing Hive-based businesses. It allows upgrading Hive tables to the Mixed-Hive format via metadata migration only, ensuring that original Hive tables remain usable and business stability is maintained.
• Paimon format: Amoro supports displaying Paimon metadata, including Schema, Options, Files, Snapshots, DDLs, and Compaction information.

An Optimizer Group (or optimizer resource group) is a concept used to divide and manage Optimizer resources. A group can contain multiple optimizers sharing the same container implementation, allowing for efficient expansion and contraction of resources based on demand.

Lifecycle Management

• Add: Create a new group via the Amoro dashboard by clicking "Add Group" on the Optimizer Groups page.
• Edit: Modify existing configurations by clicking the edit button on the Optimizer Groups page.
• Remove: Delete a group by clicking the remove button. Note: A group can only be removed if it is not referenced by any catalog or table and no optimizers belonging to that group are currently running.

Amoro tables utilize two distinct storage types to bridge the gap between offline data warehouses and real-time needs:

1. FileStore: Stores massive amounts of full historical data.
2. LogStore: Stores real-time incremental data, providing sub-second to minute-level data visibility.

LogStore connects to external message queuing middleware (currently supporting Kafka and Pulsar) to allow users to query a single table while ensuring eventual consistency between real-time and historical data sources. This eliminates the need for developers to manage both HDFS and Kafka separately.

Amoro allows you to manage configurations at three distinct levels. When a conflict occurs, the priority is applied in the following order (highest to lowest):

1. Engine: Used for engine-specific tuning (e.g., Spark or Flink).
2. Table: Specified during Create Table or modified via Alter Table operations.
3. Catalog: Used to set default values for all tables within a catalog (e.g., setting default Self-optimizing properties).

To set global defaults, configure Catalog properties. To apply specific settings to a single table, use table-level configurations.

Amoro's metrics system uses two primary types to measure table management behaviors and resource usage:

• Gauge: Provides a snapshot value of a specific metric at a given point in time (e.g., current memory usage or current status).
• Counter: A cumulative metric that increments or decrements to track totals over time (e.g., total number of failed processes since the AMS service started).

These metrics can be reported to external systems like Prometheus by configuring a metric reporter.

Amoro is designed as an out-of-the-box data lake system that uses different table formats as storage engines. This allows users to leverage the specific features of various formats while benefiting from Amoro's management, performance, and stability improvements.

Table formats in Amoro define the relationship between tables, snapshots, and files, providing MVCC (Multi-Version Concurrency Control), ACID compliance, and transaction capabilities to data lakes. They also enable advanced features like schema evolution, hidden partitioning, and data skipping.

Amoro's architecture is built around three primary pillars that provide Lakehouse management and extensibility:

• AMS (Amoro Management Service): The central service providing Lakehouse management features such as self-optimizing and data expiration. It also offers a unified catalog service for all compute engines, which can be integrated with existing metadata services.
• Plugins: A selection of external plugins for specific scenarios:
  • Optimizers: An asynchronous execution engine plugin that performs operations like merging, sorting, deduplication, and layout optimization across all table format types.
  • Terminal: SQL command-line tools with implementations like local Spark and Kyuubi.
  • LogStore: Enables millisecond to second-level SLAs for real-time data processing using message queues like Kafka and Pulsar.