repository·dev·Indexed 27 days ago
https://github.com/apache/seatunnelA multimodal, high-performance, distributed data integration tool for synchronizing structured, unstructured, and binary data. It supports execution via the native SeaTunnel Zeta Engine, Spark, and Flink. The ecosystem includes the SeaTunnel CLI for generating configurations using natural language via LLM providers (AWS Bedrock, Anthropic, OpenAI) and a multi-agent pipeline for configuration validation and repair.
EdgeSocket line protocol.SeaTunnel Engine is the default data synchronization engine for Apache SeaTunnel. It is designed for high-throughput, low-latency, and strong-consistent synchronous job operations. Key design characteristics include:
Pipeline as the minimum granularity for checkpoints and fault tolerance. Task failures are isolated to their specific pipeline, preventing entire job failures. Supports data caching to prevent data loss from source expiration during target failures.seatunnel-core-starter module serves as the foundational base module for developing new SeaTunnel connector APIs. It provides the core infrastructure required to integrate new data sources or sinks into the SeaTunnel ecosystem.A SeaTunnel job is defined as a pipeline consisting of three main components:
You can choose from different execution engines depending on your infrastructure:
| Engine | Best starting point | When to use it |
|---|---|---|
| SeaTunnel Engine (Zeta) | Recommended for most new users | When you want the simplest path to run SeaTunnel jobs end to end |
| Apache Flink | Good for existing Flink users | When you already operate Flink and want SeaTunnel to fit that platform |
| Apache Spark | Good for existing Spark users | When you already run Spark for batch workloads and want to reuse that stack |
The Vertica Sink connector allows writing data to Vertica via JDBC.
Key Capabilities:
is_exactly_once=false.CosFile sink connector allows you to output data to the Tencent Cloud COS file system. It supports multimodal data integration (e.g., videos, pictures) and ensures exactly-once delivery using 2PC commit by default. Supported file formats include text, csv, parquet, orc, json, excel, xml, binary, canal_json, debezium_json, and maxwell_json.SeaTunnel decouples state management from the underlying storage implementation through the CheckpointStorage abstraction. This allows you to choose the most suitable storage for your deployment, such as:
The system supports pluggable serialization, incremental checkpoints to reduce overhead, and automatic state cleanup.
SeaTunnel separates control logic from data processing to improve fault tolerance and scalability:
This separation allows the system to distinguish between 'control state' (which splits are assigned/pending) and 'execution progress' (the specific offset or position within a split), enabling precise recovery after failures.
The Clickhouse source connector supports parallel reading through two primary modes:
table_path parameter. Parallelism is implemented based on the table's part files, retrieved from the system.parts table.sql parameter. Parallelism is implemented via the execution of local table-based queries on each cluster shard.sql specifies a distributed table, it is automatically converted to query local tables.sql specifies a local table, the node configured in host is used as the shard.Note: If both table_path and sql are provided, the connector operates in SQL mode, using table_path primarily for metadata identification.
SeaTunnel Engine (Zeta) is a lightweight, high-performance native execution engine designed for data synchronization. It uses a Master-Worker architecture with slot-based resource management and supports checkpoint-based recovery for exactly-once semantics.