DOCSEARCHbyAlgolia
Submit Docs

mysql-binlog-connector-java

repository·master·Indexed 20 days ago

https://github.com/osheroff/mysql-binlog-connector-java

A Java connector for reading MySQL binary logs, enabling developers to tap into the MySQL replication stream via BinaryLogClient or read local binlog files using BinaryLogFileReader. It supports GTID resolution, resumable disconnects, TLS/SSL, and MariaDB compatibility including ANNOTATE_ROWS events. The library uses EventDeserializer to convert raw streams into Event objects consisting of an EventHeader and EventData.

Tokens
2.5K
Snippets
9
Records
11
Agent score
23%

What's inside mysql-binlog-connector-java

  1. Understand the API entry points and event model

    master

    The library provides two primary ways to consume MySQL binary logs depending on your use case:

    1. BinaryLogClient: Use this to read binary logs directly from a live MySQL server.
    2. BinaryLogFileReader: Use this for offline processing of existing binary log files.

    Both entry points use an EventDeserializer to convert the raw stream into Event objects. Each Event is composed of:

    • EventHeader: Contains metadata such as the EventType (defaults to using EventHeaderV4Deserializer).
    • EventData: The actual payload of the event.

    If the library encounters an event type for which no specific EventDataDeserializer is registered, it falls back to a NullEventDataDeserializer to prevent the stream from breaking.

    // Conceptual usage of the two entry points
// BinaryLogClient for live streaming
BinaryLogClient client = new BinaryLogClient("host", port, "user", "password");

// BinaryLogFileReader for offline files
BinaryLogFileReader reader = new BinaryLogFileReader(new File("path/to/binlog"));

  2. MariaDB support and ANNOTATE_ROWS

    master

    The BinaryLogClient works with MariaDB, but note two differences:

    1. MariaDB GTIDs are strings but follow a different parsing format.
    2. MariaDB can send ANNOTATE_ROWS events, which allow you to recover the original SQL used to generate rows. To enable this, use client.setUseSendAnnotateRowsEvent(true).

  3. How to identify tables from row events

    master

    Row-based replication events (like WriteRowsEventData, UpdateRowsEventData, or DeleteRowsEventData) do not contain the table name directly. Instead, they are preceded by a TableMapEventData event which contains the schema and table name.

    To map column names and types to these events, you must query the MySQL INFORMATION_SCHEMA.COLUMNS table, as this metadata is not included in the binary log.

  4. Build and test the project

    master

    To develop on this project, clone the repository and use Maven to build and test. The project uses Maven for its lifecycle.

    git clone https://github.com/shyiko/mysql-binlog-connector-java.git
cd mysql-binlog-connector-java
mvn
    git clone https://github.com/shyiko/mysql-binlog-connector-java.git
cd mysql-binlog-connector-java
mvn

  6. Use SSL/TLS for secure communication

    master

    To enable secure communication, configure the JVM system properties for your truststore and keystore, then set the SSLMode on the BinaryLogClient.

    Requirements:

    • TLSv1.1 & TLSv1.2 require JDK 7+.
    • Ensure your MySQL server is configured with SSL support (check via show global variables like 'have_%ssl';).
    System.setProperty("javax.net.ssl.trustStore", "/path/to/truststore.jks");
System.setProperty("javax.net.ssl.trustStorePassword","truststore.password");
System.setProperty("javax.net.ssl.keyStore", "/path/to/keystore.jks");
System.setProperty("javax.net.ssl.keyStorePassword", "keystore.password");

BinaryLogClient client = ...
client.setSSLMode(SSLMode.VERIFY_IDENTITY);

  8. Tap into the MySQL replication stream with BinaryLogClient

    master

    To stream real-time replication events from a MySQL server, use BinaryLogClient.

    Prerequisites:

    • The MySQL user must have REPLICATION SLAVE privileges.
    • If you do not manually specify binlogFilename and binlogPosition, the user also needs REPLICATION CLIENT privileges for automatic resolution.

    Key behaviors:

    • client.connect() is a blocking call that listens for events in the current thread.
    • To spawn a separate thread for listening, use client.connect(timeout).
    • By default, the client starts from the current master binlog position. Use client.setBinlogFilename(filename) and client.setBinlogPosition(position) to start from a specific point.
    BinaryLogClient client = new BinaryLogClient("hostname", 3306, "username", "password");
EventDeserializer eventDeserializer = new EventDeserializer();
eventDeserializer.setCompatibilityMode(
    EventDeserializer.CompatibilityMode.DATE_AND_TIME_AS_LONG,
    EventDeserializer.CompatibilityMode.CHAR_AND_BINARY_AS_BYTE_ARRAY
);
client.setEventDeserializer(eventDeserializer);
client.registerEventListener(new EventListener() {

    @Override
    public void onEvent(Event event) {
        ...
    }
});
client.connect();

  9. Read a local binary log file

    master

    Use BinaryLogFileReader to parse an existing binary log file on disk. You should use an EventDeserializer to control how event data is converted, and ensure you close the reader in a finally block.

    File binlogFile = ...
EventDeserializer eventDeserializer = new EventDeserializer();
eventDeserializer.setCompatibilityMode(
    EventDeserializer.CompatibilityMode.DATE_AND_TIME_AS_LONG,
    EventDeserializer.CompatibilityMode.CHAR_AND_BINARY_AS_BYTE_ARRAY
);
BinaryLogFileReader reader = new BinaryLogFileReader(binlogFile, eventDeserializer);
try {
    for (Event event; (event = reader.readEvent()) != null; ) {
        ...
    }
} finally {
    reader.close();
}

  10. Expose BinaryLogClient via JMX

    master

    You can register BinaryLogClient and BinaryLogClientStatistics as MBeans to monitor the client and its real-time statistics (like disconnect counts and skipped events) via JMX.

    MBeanServer mBeanServer = ManagementFactory.getPlatformMBeanServer();

BinaryLogClient binaryLogClient = ...
ObjectName objectName = new ObjectName("mysql.binlog:type=BinaryLogClient");
mBeanServer.registerMBean(binaryLogClient, objectName);

// following bean accumulates various BinaryLogClient stats
// (e.g. number of disconnects, skipped events)
BinaryLogClientStatistics stats = new BinaryLogClientStatistics(binaryLogClient);
ObjectName statsObjectName = new ObjectName("mysql.binlog:type=BinaryLogClientStatistics");
mBeanServer.registerMBean(stats, statsObjectName);

  11. Configure event deserialization with EventDeserializer

    master

    You can customize how specific event types are deserialized using EventDeserializer.setEventDataDeserializer. This is useful for skipping unnecessary events, handling unsupported types, or providing custom logic for specific EventTypes.

    EventDeserializer eventDeserializer = new EventDeserializer();

// do not deserialize EXT_DELETE_ROWS event data, return it as a byte array
eventDeserializer.setEventDataDeserializer(EventType.EXT_DELETE_ROWS,
    new ByteArrayEventDataDeserializer());

// skip EXT_WRITE_ROWS event data altogether
eventDeserializer.setEventDataDeserializer(EventType.EXT_WRITE_ROWS,
    new NullEventDataDeserializer());

// use custom event data deserializer for EXT_DELETE_ROWS
eventDeserializer.setEventDataDeserializer(EventType.EXT_DELETE_ROWS, 
    new EventDataDeserializer() {
        ...
    });

BinaryLogClient client = ...
client.setEventDeserializer(eventDeserializer);