DOCSEARCHbyAlgolia
Submit Docs

DataX High-Performance Data Synchronization Framework

repository·master·Indexed 12 days ago

https://github.com/alibaba/datax

An open-source data synchronization framework designed to move data between heterogeneous sources using a plugin-based architecture of Readers and Writers. It supports a wide range of data channels including RDBMS (MySQL, Oracle, PostgreSQL), NoSQL (HBase, MongoDB, OTS), Big Data systems (HDFS, Hive, ODPS), and others like Apache Kudu 1.11. Includes tools like datax-example for local plugin debugging and specialized plugins such as TableStoreStreamReader for incremental data export from Alibaba Cloud TableStore.

Tokens
133.4K
Snippets
196
Records
402
Agent score
94%

What's inside DataX

  2. Overview of DataX GDBWriter

    master
    GDBWriter is a DataX plugin designed to write data into Graph Database (GDB) instances. It functions by using a Gremlin Client to connect to remote GDB instances, consuming data from a DataX Reader, generating appropriate DSL statements, and executing them to write the data into the graph.

  3. Overview of TableStoreStreamReader for incremental data export

    master
    The TableStoreStreamReader is a DataX data channel designed for incremental data export from Alibaba Cloud TableStore (OTS). It supports different reading modes (Column mode and Row mode) and provides specific configurations for data type conversion and field mapping to facilitate seamless data movement from TableStore to other destinations.

  4. Introduction to DataX DatabendWriter

    master

    The databendwriter is a DataX plugin designed to write data from DataX records into Databend tables. It utilizes the databend JDBC driver and communicates via the RESTful HTTP protocol to execute queries on both open-source Databend and Databend Cloud.

    Write Mechanism: During each write batch, the plugin uploads batch data into an internal S3 stage and executes the corresponding INSERT SQL to load the data into the target Databend table.

    Best Practices: If using the Databend community distribution, it is recommended to use S3, MinIO, or OSS as the underlying storage layer. These support presigned upload operations, which helps avoid unnecessary data transfer costs.

  5. Overview of DataX

    master
    DataX is an open-source offline data synchronization tool and platform, widely used within Alibaba Group. It is the open-source version of Alibaba Cloud's DataWorks Data Integration. DataX enables efficient data synchronization between various heterogeneous data sources, including MySQL, Oracle, OceanBase, SqlServer, Postgre, HDFS, Hive, ADS, HBase, TableStore (OTS), MaxCompute (ODPS), Hologres, DRDS, and databend.

  6. What is DataX 3.0?

    master

    DataX is an open-source offline data synchronization tool designed for heterogeneous data sources. It enables stable and efficient data transfer between various sources such as relational databases (MySQL, Oracle, etc.), HDFS, Hive, ODPS, HBase, and FTP.

    Its core design philosophy uses a star topology rather than a complex mesh. DataX acts as the central transport carrier; to add a new data source, you only need to connect it to DataX, allowing seamless synchronization with all existing data sources.

  7. Overview of TSDBReader plugin

    master
    The TSDBReader plugin is designed to read data from Alibaba Cloud Time Series Database (TSDB). TSDB is a database service optimized for efficient read/write, compressed storage, and real-time computing of time-series data, commonly used for IoT and internet-scale real-time monitoring and alerting.

  8. Overview of obhbasewriter plugin

    master

    The obhbasewriter plugin is used to write data into ObHBase. It implements this by using the HBase Java client to connect to a remote HBase service and performing put operations.

    Key Features:

    • Supported Versions: OceanBase 3.x and 4.x.
    • Custom Rowkeys: Supports constructing ObHBase rowkeys by concatenating multiple source fields using the rowkeyColumn configuration.
    • Timestamp (Version) Support: Allows specifying the version/timestamp in three ways: using the current time, using a specific column from the source, or using a fixed timestamp value.

  9. Overview of OTSReader extraction modes

    master

    The OTSReader plugin is designed to read data from OTS (Table Store) and supports incremental data extraction by allowing users to specify a data range. It currently supports three extraction modes:

    1. Full Table Extraction (全表抽取): Reads the entire table.
    2. Range Extraction (范围抽取): Reads data within a specific range.
    3. Specified Partition Extraction (指定分片抽取): Reads data from specific partitions.

    This version also supports reading multi-version data while maintaining compatibility with older configuration files.

  10. Overview of hbase20xsqlwriter

    master

    hbase20xsqlwriter is a DataX plugin designed for bulk importing data into Phoenix SQL tables (built on HBase).

    Unlike standard HBase APIs that require manual data encoding for rowkeys, this plugin uses the Phoenix QueryServer's lightweight client driver to execute UPSERT statements. This approach ensures that data is written via the high-level SQL interface, allowing for automatic synchronization of all associated index tables.

    Key Features

    • Supports importing data into tables with indexes (automatically updates index tables).

    Requirements and Limitations

    • Versions: Requires Phoenix 5.x and HBase 2.x.
    • Service: Requires the Phoenix QueryServer to be running.
    • Table Type: Only supports tables created via Phoenix; does not support native HBase tables.
    • Operations: Does not support clearing existing table data or importing data with timestamps.

  11. Overview of DataX HdfsReader

    master

    HdfsReader is a DataX plugin designed to read data from the Hadoop Distributed File System (HDFS) and convert it into the DataX transmission protocol for delivery to a Writer.

    Supported File Formats:

    • textfile (text)
    • orcfile (orc)
    • rcfile (rc)
    • sequence file (seq)
    • csv (logical 2D table)

    Requirements:

    • Requires JDK 1.7 or higher.
    • Files must represent a logical 2D table.
    • For Kerberos authentication, the user's Hadoop cluster version should match the HdfsReader's Hadoop version (tested with Hadoop 2.7.1) to ensure compatibility.