GORM DBResolver

repository·master·Indexed 17 days ago

https://github.com/go-gorm/dbresolver

A GORM plugin that enables multi-database support, including read/write splitting, load balancing across replicas, and automatic connection switching based on models or tables. It provides features such as custom load balancing policies (Random, RoundRobin, StrictRoundRobin), manual connection overrides via GORM Clauses, and a specialized logger to identify resolver modes in SQL logs.

Tokens
3.6K
Snippets
13
Records
17
Agent score
67%

What's inside dbresolver

  1. Configure Load Balancing Policy

    master

    DBResolver supports load balancing for both Sources and Replicas via the Policy field in dbresolver.Config.

    The Policy must implement the following interface:

    type Policy interface {
    	Resolve([]gorm.ConnPool) gorm.ConnPool
    }

    Currently, dbresolver.RandomPolicy{} is the only implemented policy and it is used by default if no policy is specified.

  2. How automatic connection switching works

    master

    DBResolver automatically selects connections based on the table or struct being used in the query:

    1. Models/Structs: If a resolver is registered for a specific struct (e.g., &User{}), DBResolver uses that configuration when that model is used.
    2. Table Names: If a resolver is registered for a table name (e.g., "orders"), it applies to queries targeting that table.
    3. Raw SQL: DBResolver extracts the table name from the SQL.
      • If the SQL starts with SELECT, it uses replicas.
      • Otherwise, it defaults to sources.

    Examples of automatic switching:

    // Using a User Resolver (e.g., db5 is replica)
    DB.Table("users").Rows()                   // replicas `db5`
    DB.Model(&User{}).Find(&AdvancedUser{})     // replicas `db5`
    DB.Exec("update users set name = ?", "jinzhu") // sources `db1` (default)
    DB.Raw("select name from users").Row().Scan(&name) // replicas `db5`
    DB.Create(&user)                           // sources `db1` (default)
    
    // Using a Global Resolver (e.g., db2 is source, db3/db4 are replicas)
    DB.Find(&Pet{})                             // replicas `db3`/`db4`
    DB.Save(&Pet{})                             // sources `db2` 
    
    // Using an Orders Resolver (e.g., db8 is replica)
    DB.Find(&Order{})                           // replicas `db8` 
    DB.Table("orders").Find(&Report{})         // replicas `db8` 
  3. Quick Start with DBResolver

    master

    To use DBResolver, install the plugin and register it with your GORM database instance using DB.Use(). You can chain multiple Register calls to define different connection configurations for the global database, specific models (structs), or specific table names (strings).

    import (
      "gorm.io/gorm"
      "gorm.io/plugin/dbresolver"
      "gorm.io/driver/mysql"
    )
    
    DB, err := gorm.Open(mysql.Open("db1_dsn"), &gorm.Config{})
    
    DB.Use(dbresolver.Register(dbresolver.Config{
      // use `db2` as sources, `db3`, `db4` as replicas
      Sources:  []gorm.Dialector{mysql.Open("db2_dsn")},
      Replicas: []gorm.Dialector{mysql.Open("db3_dsn"), mysql.Open("db4_dsn")},
      // sources/replicas load balancing policy
      Policy: dbresolver.RandomPolicy{},
      // print sources/replicas mode in logger
      ResolverModeReplica: true,
    }).Register(dbresolver.Config{
      // use `db1` as sources (DB's default connection), `db5` as replicas for `User`, `Address`
      Replicas: []gorm.Dialector{mysql.Open("db5_dsn")},
    }, &User{}, &Address{}).Register(dbresolver.Config{
      // use `db6`, `db7` as sources, `db8` as replicas for `orders`, `Product`, `secondary`
      Sources:  []gorm.Dialector{mysql.Open("db6_dsn"), mysql.Open("db7_dsn")},
      Replicas: []gorm.Dialector{mysql.Open("db8_dsn")},
    }, "orders", &Product{}, "secondary"))
  4. Using DBResolver with Transactions

    master

    When a transaction is active, DBResolver maintains the transaction connection and will not switch between sources and replicas based on the standard configuration.

    However, you can specify which connection type to use before starting the transaction. This determines which connection the transaction is initialized with.

    // Start transaction based on default replicas db
    tx := DB.Clauses(dbresolver.Read).Begin()
    
    // Start transaction based on default sources db
    tx := DB.Clauses(dbresolver.Write).Begin()
    
    // Start transaction based on `secondary`'s sources
    tx := DB.Clauses(dbresolver.Use("secondary"), dbresolver.Write).Begin()
  5. Identify Resolver Modes in logs

    master

    When using NewResolverModeLogger, the logger inspects the context.Context for a specific key (dbresolver:resolver_mode_key). If a ResolverMode is found, the logged SQL will be prefixed with that mode in brackets.

    Supported modes:

    • source: Indicates the query is running against the primary/writer node.
    • replica: Indicates the query is running against a read replica.

    Note: If the mode is not found in the context (which can happen during transactions or if manual mode marking is not enabled), the SQL will be logged without the prefix.

  6. Configure Connection Pool settings

    master

    You can configure the underlying connection pool settings (like idle time and max connections) directly on the registered resolver using the following methods:

    • SetConnMaxIdleTime(time.Duration)
    • SetConnMaxLifetime(time.Duration)
    • SetMaxIdleConns(int)
    • SetMaxOpenConns(int)
    DB.Use(
      dbresolver.Register(dbresolver.Config{ /* xxx */ }).
      SetConnMaxIdleTime(time.Hour).
      SetConnMaxLifetime(24 * time.Hour).
      SetMaxIdleConns(100).
      SetMaxOpenConns(200)
    )
  7. Manual connection switching with Clauses

    master

    You can override the automatic connection selection using GORM Clauses. This is useful for forcing a read from a source or selecting a specific named resolver.

    • Force Write Mode: Use dbresolver.Write to ensure the query uses sources even if it is a read operation.
    • Specify Resolver: Use dbresolver.Use("name") to target a specific resolver configuration by its name.
    • Combine Both: Use both to target a specific resolver's sources.
    // Use Write Mode: read user from sources `db1`
    DB.Clauses(dbresolver.Write).First(&user)
    
    // Specify Resolver: read user from `secondary`'s replicas: db8
    DB.Clauses(dbresolver.Use("secondary")).First(&user)
    
    // Specify Resolver and Write Mode: read user from `secondary`'s sources: db6 or db7
    DB.Clauses(dbresolver.Use("secondary"), dbresolver.Write).First(&user)
  8. Wrap a GORM logger with NewResolverModeLogger

    master

    To see which connection mode (Source or Replica) is being used in your SQL logs, wrap your existing GORM logger using NewResolverModeLogger. This decorator intercepts the trace calls and prefixes the SQL string with the current ResolverMode (e.g., [source] SELECT ... or [replica] SELECT ...) if the mode is present in the context.

    This is particularly useful for debugging read/write splitting and verifying that queries are hitting the intended database nodes.

    import (
        "gorm.io/gorm/logger"
        "github.com/go-gorm/go-gorm/dbresolver"
    )
    
    // Assuming 'baseLogger' is your existing gorm.logger.Interface
    resolverLogger := dbresolver.NewResolverModeLogger(baseLogger)
    
    // Use this logger when initializing your GORM DB instance
    db, err := gorm.Open(dialector, &gorm.Config{
        Logger: resolverLogger,
    })
  9. Map specific tables to a resolver

    master

    When calling Register or Register method, you can pass optional datas ...interface{} arguments to target specific tables with a configuration.

    If a string is passed, it is treated as a direct key. If a model or other type is passed, the resolver attempts to parse it to identify the associated table name. This allows you to have different connection pools for different tables (e.g., a high-traffic logs table on a separate cluster).

    // Register a specific configuration for the 'users' table
    dbresolver.Register(dbresolver.Config{
        Sources: []gorm.Dialector{mysql.Open(userDSN)},
    }, "users")
    
    // Or using a model
    dbresolver.Register(dbresolver.Config{
        Sources: []gorm.Dialector{mysql.Open(userDSN)},
    }, &User{})
  10. Register a DBResolver for a GORM DB instance

    master

    You can register a DBResolver either via the package-level Register function or by calling the Register method on an existing *DBResolver instance.

    To apply the resolver to your GORM connection, you must call Initialize(db) where db is your *gorm.DB instance. This step is critical as it registers the necessary callbacks with GORM to enable connection switching.

    resolver := dbresolver.Register(dbresolver.Config{
        Sources: []gorm.Dialector{mysql.Open(dsn)},
    })
    
    // Initialize the resolver with your GORM DB instance
    err := resolver.Initialize(db)
    if err != nil {
        panic(err)
    }
  11. Implement custom load balancing with the Policy interface

    master

    The Policy interface allows you to define custom logic for selecting a connection pool from a slice of available pools. To implement a custom policy, you can either satisfy the Policy interface directly or use the PolicyFunc type to wrap a function with the signature func([]gorm.ConnPool) gorm.ConnPool.

    type Policy interface {
    	Resolve([]gorm.ConnPool) gorm.ConnPool
    }
    
    type PolicyFunc func([]gorm.ConnPool) gorm.ConnPool
    
    func (f PolicyFunc) Resolve(connPools []gorm.ConnPool) gorm.ConnPool {
    	return f(connPools)
    }
  12. Configure DBResolver with Sources and Replicas

    master

    To set up read/write splitting or multiple database connections, use the Config struct with the Register method.

    • Sources: A slice of gorm.Dialector representing the primary (write) databases.
    • Replicas: A slice of gorm.Dialector representing the read-only replica databases.
    • Policy: Determines how replicas are selected (defaults to RandomPolicy if not specified).
    • TraceResolverMode: If set to true, enables logging for the resolver mode.

    You can register multiple configurations. If datas are provided, they can be used to map specific tables or strings to a specific resolver configuration.

    dbresolver.Register(dbresolver.Config{
        Sources: []gorm.Dialector{postgres.Open(writeDSN)},
        Replicas: []gorm.Dialector{postgres.Open(replicaDSN1), postgres.Open(replicaDSN2)},
        Policy: dbresolver.RandomPolicy{},
    })