repository·master·Indexed 26 days ago
https://github.com/dtstack/chunjunChunJun (formerly FlinkX) is a distributed integration framework based on Apache Flink for data synchronization and calculation between heterogeneous data sources such as MySQL, Oracle, Hive, and Kudu using JSON templates or SQL scripts. It supports multiple deployment modes including Local, Standalone, YARN (Per-Job and Session), and Kubernetes (Application and Session).
The HDFS Source plugin allows reading from and writing to HDFS paths for text, orc, and parquet file types. It is commonly used with Hive tables by reading/writing data directly to the HDFS paths corresponding to Hive partitions.
Important Limitations:
The HDFS Sink plugin allows reading from and writing to HDFS paths using TextFile, Orc, or Parquet formats. It is commonly used with Hive tables by writing data directly to the HDFS paths corresponding to Hive partitions.
Key Behaviors:
checkpoint is enabled, it uses a two-phase commit mechanism: data is first generated in a .data directory, then copied to the formal directory and marked. In the commit phase, marked files in .data are deleted; in the rollback phase, marked files in the formal directory are deleted.The Hive sink plugin enables real-time writing of data into Hive tables. It supports automatic table creation and automatic partition creation based on the current system time. It also supports dynamic table name parsing and group mapping to route different data to different Hive tables based on mapping rules.
Key Features:
partitionType.analyticalRules to resolve table names from the incoming data stream.checkpoint to be enabled. ChunJun uses a two-phase commit mechanism: data is first written to a .data directory, then copied to the official directory during the commit phase. This ensures data is only visible in Hive after a successful checkpoint.The GBase Lookup connector is used for GBase dimension table lookups. It supports two caching modes:
ALL): Loads the entire dimension table into memory. Recommended for small datasets that do not change frequently.LRU): Queries data asynchronously and uses an LRU (Least Recently Used) cache in memory. Recommended for large datasets.Supported GBase Version:
Plugin Names:
gbase-xgbase-xLogMiner is an Oracle tool used to parse RedoLog (and archived log) files to extract specific data changes as SQL statements. It is composed of PL/SQL packages and dynamic views.
Key concepts:
v$log and v$logfile).v$archived_log).V$LOGMNR_CONTENTS view, which can be queried via SQL to obtain real-time data.The ElasticSearch Sink plugin allows you to write data to a specified Elasticsearch index. It supports Elasticsearch version 7.x.
There are two plugin types available depending on your usage mode:
elasticsearch7writer.elasticsearch7-x.The Postgres CDC plugin captures real-time change data from PostgreSQL. Note that the current sink plugin does not support data restoration; it only writes the captured change log data.
Supported Versions:
protobuf-x format allows reading and writing Protocol Buffer data using Flink SQL. It supports both Protocol Buffer 2.x and 3.x versions.A complete ChunJun job script configuration consists of two main sections: content and setting.
content: Configures the data source and destination. It contains a list of reader and writer configurations.setting: Configures the global environment for the task, including concurrency, error handling, metrics, restoration, and logging.Basic structure:
{
"job": {
"content": [
{
"reader": {},
"writer": {}
}
],
"setting": {
"speed": {},
"errorLimit": {},
"metricPluginConf": {},
"restore": {},
"log": {},
"dirty": {}
}
}
}{
"job": {
"content": [
{
"reader": {},
"writer": {}
}
],
"setting": {
"speed": {},
"errorLimit": {},
"metricPluginConf": {},
"restore": {},
"log": {},
"dirty": {}
}
}
}