Moonlink Managed Ingestion Engine

Supported Ingestion Sources (Write Paths)

• PostgreSQL CDC: Via logical replication (millisecond latency).
• REST API: HTTP endpoint for direct event ingestion.
• Kafka: Sink support is currently in the roadmap.
• OTEL: Sink support is currently on the roadmap.

Supported Query Engines (Read Path)

Moonlink commits data as Iceberg v3 tables with deletion vectors, which can be queried by:

• DuckDB
• Apache Spark
• Postgres (using pg_duckdb or pg_mooncake extensions)

Real-Time Reads (Sub-second freshness)

For workloads requiring immediate visibility, use these specific extensions:

• DuckDB: via duckdb_mooncake extension.
• Postgres: via pg_mooncake extension.
• DataFusion: via Moonlink Datafusion.

The Moonlink chaos test suite is a stress-testing tool designed to validate the core engine's persistence and snapshot semantics. It simulates adversarial event sequences (appends, updates, deletes, and transaction types) interleaved with maintenance operations (index merges, data compaction, and snapshots) across various backends like FS, S3, and GCS.

It specifically validates:

• Causality & ordering: Ensuring LSNs strictly increase and transaction boundaries (Begin/End) are respected.
• State consistency: Ensuring only committed changes affect snapshots.
• Persistence fidelity: Verifying that a snapshot at a specific LSN can be reloaded to exactly match the in-memory state.
• Maintenance safety: Ensuring maintenance runs only during quiescent periods.

Moonlink is an ingestion engine designed to prevent the common issues of real-time Iceberg ingestion, such as tiny data files and metadata explosion. It achieves this through two primary write paths:

1. Inserts: Incoming raw inserts are buffered in Arrow format and then flushed as size-tuned Parquet files to Iceberg.
2. Deletes: Deletes are mapped to deletion vectors using an index built on row positions, rather than creating expensive equality-deletes.

By buffering, caching, and indexing data on NVMe, Moonlink ensures that Iceberg tables remain read-optimized without requiring manual compaction jobs.

To generate profiling data for a benchmark, use the --profile-time flag passed through cargo to the underlying benchmarking tool. This allows you to generate flamegraphs to analyze performance bottlenecks.

After running the command, the resulting flamegraph can be found in target/criterion/TEST_NAME/profile, where TEST_NAME is the name of the benchmark you ran.

cargo bench --bench microbench_write_mooncake_table --features='bench' -- --profile-time=5

Moonlink provides microbenchmarks to measure the performance of its core components. You can run all benchmarks or target a specific benchmark using cargo bench. Note that the bench feature must be enabled.

# Run all benchmarks
cargo bench --features='bench'

# Run a specific benchmark (e.g., microbench_write_mooncake_table)
cargo bench --bench microbench_write_mooncake_table --features='bench'

To deploy Moonlink to a local Kubernetes cluster using kind, ensure you have Docker, kind, kubectl, and yq installed. Use the provided setup script to automate cluster creation, image loading, and manifest application.

Prerequisites Verification:

docker --version && kind --version && kubectl version --client=true --output=yaml && yq --version

Deployment Steps:

1. Run the setup script to create/ensure the cluster and namespace, build/load the moonlink:dev image, and apply manifests.
2. Verify the deployment status using kubectl.

# 1) Create/ensure cluster & namespace, build & load image, deploy manifests, wait for rollout
./deploy/kind/local_setup_script.sh

# 2) Verify deployment status
kubectl get pods,svc -n moonlink

To get started with Moonlink, you need to build the service from source and run it with a local data directory.

1. Clone and Build: Clone the repository and use cargo to build the moonlink_service binary.
2. Start Service: Run the binary and provide a path for data storage (e.g., ./data).
3. Verify: Use curl to check the /health endpoint to ensure the service is running.

# 1. Clone & Build
git clone https://github.com/Mooncake-Labs/moonlink.git
cd moonlink
cargo build --release --bin moonlink_service

# 2. Start the Moonlink Service
./target/release/moonlink_service ./data

# 3. Verify Service Health
curl http://localhost:3030/health

Use the cleanup.sh script to remove Moonlink resources. By default, it only deletes the resources defined in the manifests. You can use environment variables to perform a more thorough cleanup.

Standard Cleanup (Resources only)

./deploy/kind/cleanup.sh

Advanced Cleanup

• Delete the entire kind cluster: Set NUKE_CLUSTER=true.
• Delete the moonlink namespace: Set NUKE_NAMESPACE=true.

# Delete the entire kind cluster
NUKE_CLUSTER=true ./deploy/kind/cleanup.sh

# Delete the 'moonlink' namespace
NUKE_NAMESPACE=true ./deploy/kind/cleanup.sh

To ingest data, send a POST request to the /ingest/{table_name} endpoint. The payload requires an operation (e.g., insert), a request_mode (e.g., async), and a data object containing the row values matching the table schema.

curl -X POST http://localhost:3030/ingest/users \
  -H "Content-Type: application/json" \
  -d '{
    "operation": "insert",
    "request_mode": "async",
    "data": {
      "id": 1,
      "name": "Alice Johnson",
      "email": "alice@example.com",
      "age": 30,
      "created_at": "2024-01-01"
    }
  }'

To ensure reproducibility of a specific sequence of events, use the --seed flag. If no seed is provided, the suite uses a timestamp-derived default. Providing a specific u64 seed allows you to recreate the exact same sequence of operations to debug failures.

To see the events even when the test passes, use the --print-events-on-success flag.

# Example of running with a specific seed for reproducibility
./chaos_test_binary --seed 123456789