go-mysql

repository·master·Indexed 26 days ago

https://github.com/go-mysql-org/go-mysql

A pure Go library for handling the MySQL network protocol and replication. It provides packages for building MySQL-compatible servers and proxies, parsing binlogs, and implementing data synchronization tools via the replication and canal packages. It includes a client driver compatible with MySQL 5.5.x through 8.0.x and supports integration with database/sql and GORM.

Tokens
5.8K
Snippets
15
Records
28
Agent score
89%

What's inside go-mysql

  1. Implement a custom MySQL server with the Server package

    master

    The server package provides a framework for building MySQL-compatible servers or proxies. It is compatible with MySQL 5.5, 5.6, 5.7, and 8.0 clients.

    server.NewConn() provides a default configuration that:

    1. Automatically generates default TLS/SSL certificates.
    2. Supports mysql_native_password, caching_sha2_password, and sha256_password (defaults to mysql_native_password).
    3. Uses an in-memory user credential provider.

    To use custom configurations, use server.NewServer() and create connections via server.NewCustomizedConn().

    package main
    
    import (
    	"log"
    	"net"
    
    	"github.com/go-mysql-org/go-mysql/server"
    )
    
    func main() {
    	// Listen for connections on localhost port 4000
    	l, err := net.Listen("tcp", "127.0.0.1:4000")
    	if err != nil {
    		log.Fatal(err)
    	}
    
    	// Accept a new connection once
    	c, err := l.Accept()
    	if err != nil {
    		log.Fatal(err)
    	}
    
    	// Create a connection with user root and an empty password.
    	conn, err := server.NewConn(c, "root", "", server.EmptyHandler{})
    	if err != nil {
    		log.Fatal(err)
    	}
    
    	// Handle commands as long as the client keeps sending them
    	for {
    		if err := conn.HandleCommand(); err != nil {
    			log.Fatal(err)
    		}
    	}
    }
  2. Build example applications from the cmd directory

    master

    The project includes several example applications in the cmd directory. You can build them by running make build in the project root. The resulting binaries will be placed in the bin/ directory.

    Available binaries:

    • go-binlogparser: parses a binlog file at a given offset
    • go-canal: streams binlog events from a server to canal
    • go-mysqlbinlog: streams binlog events
    • go-mysqldump: like mysqldump, but in Go
    • go-mysqlserver: fake MySQL server
    make build
  3. Migrate from siddontang/go-mysql to go-mysql-org/go-mysql

    master

    To migrate your project to the new repository, add a replace directive to your go.mod file to point the old import path to the new one.

    go mod edit -replace=github.com/siddontang/go-mysql=github.com/go-mysql-org/go-mysql@v1.13.0

    Or manually edit go.mod:

    replace github.com/siddontang/go-mysql => github.com/go-mysql-org/go-mysql v1.13.0
  4. Use go-mysql with database/sql

    master

    You can use go-mysql as a standard Go database driver by importing the github.com/go-mysql-org/go-mysql/driver package with a blank identifier. The connection is established using a DSN (Data Source Name) in the format user:password@addr/dbname?param=value via sql.Open("mysql", dsn).

    package main
    
    import (
    	"database/sql"
    
    	_ "github.com/go-mysql-org/go-mysql/driver"
    )
    
    func main() {
    	// dsn format: "user:password@addr?dbname"
    	dsn := "root@127.0.0.1:3306?test"
    	db, _ := sql.Open("mysql", dsn)
    	db.Close()
    }
  5. Use the Client package for MySQL connections

    master

    The client package provides a simple driver for communicating with MySQL servers. It supports MySQL versions 5.5.x through 8.0.x. You can use a connection pool for managing multiple connections efficiently.

    import (
        "github.com/go-mysql-org/go-mysql/client"
    )
    
    // Create a connection pool
    pool := client.NewPool(log.Debugf, 100, 400, 5, "127.0.0.1:3306", `root`, ``, `test`)
    
    // Get a connection from the pool
    conn, _ := pool.GetConn(ctx)
    defer pool.PutConn(conn)
    
    // Execute commands
    conn.Execute() 
    // or conn.Begin() etc...
  6. Change the driver name for GORM compatibility

    master

    To use go-mysql with GORM, you can change the driver name from mysql to gomysql using build-time -ldflags.

    # Build with custom driver name
    go build -ldflags '-X "github.com/go-mysql-org/go-mysql/driver.driverName=gomysql"'

    Then use it in GORM:

    import (
      _ "github.com/go-mysql-org/go-mysql/driver"
      "gorm.io/driver/mysql"
      "gorm.io/gorm"
    )
    
    db, err := gorm.Open(mysql.New(mysql.Config{
      DriverName: "gomysql",
      DSN: "gorm:gorm@127.0.0.1:3306/test",
    }))
  7. Implement MySQL replication with the replication package

    master

    The replication package allows you to act as a MySQL replica to sync binlogs from a master server. You can use this to trigger actions like updating caches or synchronizing data. You can sync using a specific binlog file and position, or via GTID (Global Transaction Identifier) replication.

    Note: When using MariaDB 11.4+, you may need to enable FillZeroLogPos to ensure accurate position tracking for events inside transactions.

    import (
    	"github.com/go-mysql-org/go-mysql/replication"
    	"os"
    )
    
    // Create a binlog syncer with a unique server id
    cfg := replication.BinlogSyncerConfig {
    	ServerID: 100,
    	Flavor:   "mysql",
    	Host:     "127.0.0.1",
    	Port:     3306,
    	User:     "root",
    	Password: "",
    }
    syncer := replication.NewBinlogSyncer(cfg)
    
    // Start sync with specified binlog file and position
    streamer, _ := syncer.StartSync(mysql.Position{binlogFile, binlogPos})
    
    for {
    	ev, _ := streamer.GetEvent(context.Background())
    	ev.Dump(os.Stdout)
    }
  8. Sync MySQL data using the Canal package

    master

    The canal package provides incremental data synchronization from MySQL to external systems like Redis or Elasticsearch. It works by dumping initial data and then processing binlog changes.

    Requirements:

    • Use ROW format for binlogs.
    • Full binlog row image is preferred to avoid errors when primary keys change during updates.
    package main
    
    import (
    	"github.com/go-mysql-org/go-mysql/canal"
    )
    
    type MyEventHandler struct {
    	canal.DummyEventHandler
    }
    
    func (h *MyEventHandler) OnRow(e *canal.RowsEvent) error {
    	log.Infof("%s %v\n", e.Action, e.Rows)
    	return nil
    }
    
    func (h *MyEventHandler) String() string {
    	return "MyEventHandler"
    }
    
    func main() {
    	cfg := canal.NewDefaultConfig()
    	cfg.Addr = "127.0.0.1:3306"
    	cfg.User = "root"
    	// We only care about table canal_test in test db
    	cfg.Dump.TableDB = "test"
    	cfg.Dump.Tables = []string{"canal_test"}
    
    	c, err := canal.NewCanal(cfg)
    	if err != nil {
    		log.Fatal(err)
    	}
    
    	// Register a handler to handle RowsEvent
    	c.SetEventHandler(&MyEventHandler{})
    
    	// Start canal
    	c.Run()
    }
  9. Configure FillZeroLogPos for MariaDB 11.4+ compatibility

    master

    MariaDB 11.4+ uses an optimization where some events have LogPos=0. To track these positions accurately, set FillZeroLogPos: true in your BinlogSyncerConfig. This enables the BINLOG_SEND_ANNOTATE_ROWS_EVENT flag and dynamic LogPos calculation. This setting only affects the mariadb flavor.

    cfg := replication.BinlogSyncerConfig {
    	ServerID: 100,
    	Flavor:   "mariadb",
    	Host:     "127.0.0.1",
    	Port:     3306,
    	User:     "root",
    	Password: "",
    	// Enable dynamic LogPos calculation for MariaDB 11.4+
    	FillZeroLogPos: true,
    }
  10. Run MySQL test environments using Docker Compose

    master
    The repository provides a docker-compose.yaml file to spin up multiple versions of MySQL for testing and development. Each service is configured with specific ports and environment variables to facilitate testing against different MySQL versions and configurations (including SSL and specific authentication plugins).
  11. Implement a custom NamedValueChecker

    master

    You can implement a custom NamedValueChecker to handle query arguments before they reach the driver. This requires a full import of the driver package. Returning sqlDriver.ErrSkip allows the driver to fall back to the default value converter.

    import (
     "database/sql"
    
     "github.com/go-mysql-org/go-mysql/driver"
    )
    
    func main() {
     driver.AddNamedValueChecker(func(nv *sqlDriver.NamedValue) error {
      rv := reflect.ValueOf(nv.Value)
      if rv.Kind() != reflect.Uint64 {
       // fallback to the default value converter when the value is not a uint64
       return sqlDriver.ErrSkip
      }
    
      return nil
     })
    
     conn, err := sql.Open("mysql", "root@127.0.0.1:3306/test")
     defer conn.Close()
    
     stmt, err := conn.Prepare("select * from table where id = ?")
     defer stmt.Close()
     var val uint64 = math.MaxUint64
     // without the NamedValueChecker this query would fail
     result, err := stmt.Query(val)
    }
  12. Extend the driver with SetDSNOptions

    master

    The driver.SetDSNOptions function allows you to define custom driver options that can be triggered via the DSN. This requires a full import of the driver package (not just a side-effect import).

    import (
     "database/sql"
    
     "github.com/go-mysql-org/go-mysql/driver"
    )
    
    func main() {
     driver.SetDSNOptions(map[string]DriverOption{
      "no_metadata": func(c *client.Conn, value string) error {
       c.SetCapability(mysql.CLIENT_OPTIONAL_RESULTSET_METADATA)
       return nil
      },
     })
    
     // dsn format: "user:password@addr/dbname?"
     dsn := "root@127.0.0.1:3306/test?no_metadata=true"
     db, _ := sql.Open(dsn)
     db.Close()
    }