DOCSEARCHbyAlgolia
Submit Docs

Canal MySQL Binlog Parsing Tool

repository·master·Indexed 12 days ago

https://github.com/alibaba/canal

An open-source MySQL binlog parsing tool for incremental data subscription and consumption. It acts as a MySQL slave to capture data changes and deliver them to downstream systems such as Kafka, RocketMQ, or custom clients. Includes components like canal-admin for management, canal-server for data capture, and a client-adapter for synchronizing data to HBase, ElasticSearch, and other relational databases via JDBC.

Tokens
15.4K
Snippets
55
Records
70
Agent score
96%

What's inside Canal

  1. What is Canal and how does it work?

    master

    Canal is a tool designed for parsing MySQL database incremental logs (binlog) to provide incremental data subscription and consumption. It is commonly used for database mirroring, real-time backups, index maintenance (e.g., building inverted indexes), cache refreshing, and incremental data processing with business logic.

    How it works

    Canal simulates a MySQL slave by using the MySQL dump protocol.

    1. MySQL Master: Writes data changes to the binary log (binlog).
    2. Canal (as a Pseudo-Slave): Sends a dump request to the MySQL master.
    3. Data Flow: The master pushes binary log events to Canal, and Canal parses these byte streams into usable data.

    Supported MySQL Versions

    • 5.1.x, 5.5.x, 5.6.x, 5.7.x, and 8.0.x

  2. How Canal's client-server architecture enables multi-language support

    master

    Canal uses a client-server architecture where the interaction protocol is based on protobuf 3.0. This allows developers to implement consumption logic in various programming languages by using different client implementations.

    Available Client Implementations

    Additionally, Canal can deliver binlog changes to Message Queue (MQ) systems like Kafka or RocketMQ, which can then be consumed by any language supported by those MQ systems.

  3. Understand the canal-adapter architecture

    master

    The client-adapter consists of two main components: Adapters and the Launcher.

    • Adapters: These are multiple fat JARs (e.g., client-adapter.hbase-1.1.1-jar-with-dependencies.jar). Each adapter contains its own dependencies and is loaded dynamically by the Launcher using the SPI (Service Provider Interface) mechanism.
    • Launcher: A SpringBoot project that starts the canal-client and provides REST management interfaces. It orchestrates the adapters based on configuration.

    Typical directory structure:

    - bin
    (startup/stop scripts)
- lib
    (adapter fat jars)
- conf
    application.yml
    (adapter specific configs, e.g., hbase/)
- logs

  5. Build the canal/osbase Docker image

    master

    To build the canal/osbase image, navigate to the amd64 directory and use docker build with the provided Dockerfile. Using the --no-cache flag ensures a clean build from the base image.

    cd amd64 && docker build --no-cache -t canal/osbase ./ -f Dockerfile

  6. Manage Canal dynamically with Canal Admin

    master

    Introduced in version 1.1.4, canal-admin provides a WebUI for the dynamic management of Canal instances. It supports online, white-box operations for:

    • Configuration management
    • Task management
    • Log monitoring

    For detailed setup and usage, refer to the Canal Admin Guide.

  7. Initialize Canal Databases

    master

    Before deploying, you must initialize the required MySQL databases using the provided SQL scripts:

    1. Canal Manager Database: Use /admin/admin-web/src/main/resources/canal_manager.sql to set up the management database.
    2. TSDB Database: Use /deployer/src/main/resources/spring/tsdb/sql/create_table.sql to set up the Time Series Database (TSDB) tables.

  8. Setup and Run RDB Data Synchronization

    master

    To run RDB synchronization:

    1. Place the target database's JDBC driver (e.g., ojdbc6.jar) into the lib folder.
    2. Configure application.yml and the mapping files in conf/rdb/.
    3. Start the adapter using the startup script:
    bin/startup.sh

    Verification: Modify data in the source MySQL table; changes should automatically sync to the target table and appear in the DML logs.

  9. Build the canal/osadmin Docker image

    master

    To build the canal/osadmin image, navigate to the amd64 directory and use docker build. Note that this build uses a specific Dockerfile named Dockerfile_admin instead of the default.

    cd amd64 && docker build --no-cache -t canal/osadmin ./ -f ./Dockerfile_admin

  10. Deploy canal-admin via Helm

    master

    Deploy the canal-admin chart by providing a values.yaml file that configures the Spring datasource and Canal admin credentials.

    After the pods are running, log in with the default credentials:

    • Username: admin
    • Password: 123456

    Then, create a cluster named default and set the ZK address (e.g., canal-zookeeper.default.svc.cluster.local:2181).

    # Example canal-admin values.yaml snippet
admin:
  config: |
    server:
      port: 8089
    spring:
      jackson:
        date-format: yyyy-MM-dd HH:mm:ss
        time-zone: GMT+8
    spring.datasource:
      address: your_db_host:3306
      database: canal_manager
      username: ****
      password: ****
      driver-class-name: com.mysql.jdbc.Driver
      url: jdbc:mysql://${spring.datasource.address}/${spring.datasource.database}?useUnicode=true&characterEncoding=UTF-8&useSSL=false&allowPublicKeyRetrieval=true
      hikari:
        maximum-pool-size: 30
        minimum-idle: 1
    canal:
      adminUser: admin
      adminPasswd: admin
    helm install canal-admin -f ./admin-values.yaml ./canal-admin

  11. Deploy canal-server via Helm

    master

    Deploy the canal-server chart using a values.yaml that points to the canal-admin instance. When canal.admin.register.auto is set to true, the server will automatically register itself with the admin cluster.

    # Example canal-server values.yaml snippet
server:
  config: |
    canal.port = 11111
    canal.metrics.pull.port = 11112
    canal.register.ip = 

    # canal admin config
    canal.admin.manager = canal-admin.default:8089
    canal.admin.port = 11110
    canal.admin.user = admin
    canal.admin.passwd = 4ACFE3202A5FF5CF467898FC58AAB1D615029441
    # admin auto register
    canal.admin.register.auto = true
    canal.admin.register.cluster = default
    helm install canal-server -f ./server-values.yaml ./canal-server

  12. Install Zookeeper for Canal High Availability

    master

    Canal uses Zookeeper as a registration center to achieve high availability. You can install a Zookeeper cluster using the Bitnami Helm chart.

    helm install canal-zookeeper oci://registry-1.docker.io/bitnamicharts/zookeeper