DOCSEARCHbyAlgolia
Submit Docs

Apache Flink CDC Documentation

repository·master·Indexed 27 days ago

https://github.com/apache/flink-cdc

A distributed data integration tool built on Apache Flink for real-time and batch data integration. It supports full database synchronization, schema evolution, and data transformation. Flink CDC provides three ways to define pipelines: a zero-code YAML API (Pipeline API), a SQL API for Flink SQL integration, and a DataStream API for custom programmatic applications. Key features include exactly-once semantics, incremental snapshot algorithms to avoid database locks, and sub-second end-to-end latency.

Tokens
186.1K
Snippets
321
Records
673
Agent score
89%

What's inside Flink CDC

  1. Overview of Flink CDC Sources

    master

    Flink CDC sources is a collection of source connectors for Apache Flink designed to ingest change data from various databases using Change Data Capture (CDC). Some connectors utilize Debezium as the underlying engine to capture data changes.

    These sources can be used via multiple API layers including the YAML API (Pipeline API), SQL API (Table/SQL API), and the DataStream API.

  2. Overview of Apache Flink CDC

    master

    Apache Flink CDC is a distributed data integration tool designed for both real-time and batch data. It simplifies data movement and transformation by allowing users to describe pipelines using a YAML-based API.

    Key capabilities include:

    • Change Data Capture (CDC): Supports distributed scanning of historical database data followed by an automatic switch to incremental change data capturing using an incremental snapshot algorithm that avoids database locks.
    • Schema Evolution: Automatically creates downstream tables by inferring structures from upstream tables and applies upstream DDL changes to downstream systems.
    • Streaming Pipeline: Runs in streaming mode by default, providing sub-second end-to-end latency for real-time binlog synchronization.
    • Data Transformation: Supports ETL operations such as column projection, computed columns, filter expressions, and scalar functions.
    • Full Database Sync: Can synchronize all tables from a source database instance to a downstream system in a single job by configuring database and table lists.
    • Exactly-Once Semantics: Ensures exactly-once processing for both historical data reading and continuous CDC event consumption, even after job failures.

  4. Use the Fluss Pipeline Connector as a Data Sink

    master

    The Fluss Pipeline Connector acts as a Data Sink to write data into Fluss. It supports automatic table creation, data synchronization, and schema change synchronization in LENIENT mode.

    Key capabilities:

    • Automatic Table Creation: Creates tables if they do not exist. Note that tables created this way do not have partition keys. The number of buckets is controlled by bucket.num, and data distribution is controlled by bucket.key.
    • Schema Change Synchronization: When configured with schema.change.behavior: LENIENT, the connector supports:
      • Adding columns: Appends the new column to the Fluss table.
      • Deleting columns: Does not actually delete the column; instead, it ignores the deletion and writes null for that column in subsequent writes.
      • Renaming columns: Converted into an

  6. Configure the Iceberg Pipeline Connector

    master

    The Iceberg Pipeline Connector acts as a Data Sink for Flink CDC pipelines, allowing you to write data to Apache Iceberg tables. It supports automatic table creation, schema synchronization (propagating column additions), and both batch and streaming data replication.

    Key Requirements

    • Primary Keys: The source table must have a primary key. Tables without a primary key are not supported.
    • Semantics: Exactly-once semantics are not supported. The connector provides at-least-once delivery combined with idempotent writes using the table's primary key.

  7. Understand Kafka Sink Topic Routing and Creation

    master

    By default, the Kafka connector writes to a topic named using the TableId string format: namespace.schemaName.tableName.

    • Automatic Creation: If the target topic does not exist, the connector will create it automatically.
    • Custom Routing: You can change the destination topic using the route function within the pipeline or by using the sink.tableId-to-topic.mapping option.

  8. Configure the StarRocks Pipeline Sink

    master

    The StarRocks connector acts as a Data Sink in a Flink CDC pipeline, enabling data synchronization, automatic table creation, and schema change synchronization. It specifically supports StarRocks primary key tables, so ensure your source tables have primary keys.

    Note that the connector does not support exactly-once semantics; it uses at-least-once delivery combined with primary key tables to achieve idempotent writes.

    sink:
  type: starrocks
  name: StarRocks Sink
  jdbc-url: jdbc:mysql://127.0.0.1:9030
  load-url: 127.0.0.1:8030
  username: root
  password: pass

pipeline:
   name: MySQL to StarRocks Pipeline
   parallelism: 2

  10. Use the Hudi Pipeline Connector as a Data Sink

    master

    The Hudi Pipeline Connector acts as a Data Sink to write data into Apache Hudi. It supports automatic Hudi table creation, automatic schema evolution synchronization, and real-time data synchronization.

    Key Constraints and Behaviors

    • Primary Keys: Source tables must have a primary key defined. You can override primary key configurations using transform rules or by setting ordering.fields in table-options.
    • Supported Types: Currently only supports MERGE_ON_READ table types and BUCKET index types.
    • Delivery Semantics: Does not support exactly-once semantics. It achieves idempotent writes using at-least-once semantics and primary key tables.

  11. Understand the Table ID concept

    master

    In Flink CDC, a Table Id is used to establish a mapping relationship between Flink and the storage objects of an external system. To ensure compatibility across various data systems, the Table Id is represented as a 3-tuple: (namespace, schemaName, tableName).

    Connectors are responsible for mapping this logical Table Id to the specific storage objects in the target system.

  12. Understand Flink CDC Event types

    master

    In Flink CDC, an Event is a specialized record in a Flink data stream representing captured changes from an external system. Events consist of a Table ID and a payload. There are two primary categories of events:

    1. DataChangeEvent

    Describes data changes in the source. It contains five fields:

    • Table ID: The identifier for the table.
    • Before: The pre-image of the data.
    • After: The post-image of the data.
    • Operation type: The type of change operation.
    • Meta: Metadata associated with the change.

    Pre-defined Operation Types:

    • Insert: New data entry (before = null, after = new data).
    • Delete: Removal of data (before = removed data, after = null).
    • Update: Modification of existing data (before = data before change, after = data after change).
    • Replace: (Note: specific details for Replace were not provided in the source).

    2. SchemaChangeEvent

    Describes changes to the table structure in the external system. Types include:

    • AddColumnEvent: A new column is added.
    • AlterColumnTypeEvent: A column's type is changed.
    • CreateTableEvent: A new table is created (also used to describe the schema for pre-emitted DataChangeEvents).
    • DropColumnEvent: A column is removed.
    • RenameColumnEvent: A column name is changed.