DBSyncer Documentation

DBSyncer is an open-source data synchronization middleware designed to facilitate data movement between various sources and targets. It supports a wide range of scenarios including relational databases (MySQL, Oracle, SQL Server, PostgreSQL), non-relational databases (Elasticsearch, Redis), message queues (Kafka), and files.

Key features include:

• Combination Driven: Map relationships between different database types (e.g., relational to non-relational).
• Real-time Monitoring: Monitor full and incremental synchronization status, results, and logs.
• Customizable Logic: Upload plugins to implement custom transformation and synchronization logic.

To install DBSyncer using the pre-compiled zip files:

1. Install JDK 1.8.
2. Download the installation package (dbsyncer-x.x.x.zip) from the releases page.
3. Extract the package.
4. Run the startup script:
   • Windows: Execute bin/startup.bat
   • Linux: Execute bin/startup.sh
5. Access the web interface at http://127.0.0.1:18686.
6. Default credentials: admin / admin.

To build DBSyncer manually, ensure you have JDK and Maven installed.

$ git clone https://gitee.com/ghi/dbsyncer.git
$ cd dbsyncer
$ chmod u+x build.sh
$ ./build.sh

Once the container is running, use these commands for management:

Log Management:

• View local logs (on host): ls -la /opt/dbsyncer/logs
• View container logs (last 20 lines): docker logs --tail 20 dbsyncer
• Follow real-time logs: docker logs -f dbsyncer
• View logs inside the container: ls -la /app/dbsyncer/logs

Container Control:

• Enter container shell: docker exec -it dbsyncer /bin/bash
• Stop container: docker stop dbsyncer
• Start container: docker start dbsyncer
• Restart container: docker restart dbsyncer
• Remove container: docker rm dbsyncer (must stop first)

# Example: Follow real-time logs
docker logs -f dbsyncer

# Example: Enter container
docker exec -it dbsyncer /bin/bash

To start the DBSyncer application on a Windows environment, execute the startup.bat script located in the bin directory. This script configures the necessary environment variables, JVM memory settings, and system properties required for the application to run correctly.

Prerequisites

• Java JDK 8: You must have JDK 8 installed.
• JAVA_HOME: The JAVA_HOME environment variable must be set and point to your JDK 8 installation directory.

Execution

Open a command prompt, navigate to the directory containing startup.bat, and run:

startup.bat

If you encounter connection errors, check the following known issues:

• MySQL Connection Failure: The default driver version is 8.0.21. If you are using MySQL 5.x, you must manually replace the driver with mysql-connector-java-5.1.40.jar.
• SQL Server SSL/TLS Errors: If you see errors regarding TLS protocol versions (e.g., The server selected protocol version TLS10 is not accepted by client preferences [TLS12]), check your driver and SSL configuration.
• Data Encoding/Garbled Text: If data becomes garbled (e.g., importing MySQL 8 tables to SQL Server 2008R2 nvarchar fields), verify character set settings between source and target.

The DBSyncer application requires a configuration file located at conf/application.properties relative to the application home directory (DBS_HOME). The startup script will fail if this file is missing.

Required File Structure:

• $DBS_HOME/conf/application.properties

When running DBSyncer in a Docker or containerized environment using the startup-docker.sh script, you can extend the default JVM configuration using the JAVA_OPTS_EXT environment variable. This is useful for overriding memory limits or adding specific performance tuning parameters.

Supported Environment Variables:

• JAVA_OPTS_EXT: A space-separated string of additional JVM arguments (e.g., -Xmx3500m -XX:MaxDirectMemorySize=512m).

export JAVA_OPTS_EXT="-Xmx3500m -XX:MaxDirectMemorySize=512m"