go-mysql-server

repository·main·Indexed 25 days ago

https://github.com/dolthub/go-mysql-server

A data-source agnostic SQL engine that uses the MySQL dialect and wire protocol. It enables developers to run queries against any data source by implementing specific interfaces, such as sql.DatabaseProvider and sql.Table, or by using a built-in in-memory implementation for testing. The engine follows a lifecycle of parsing via vitess, analysis for optimization, execution of plan nodes, and communication via the MySQL wire protocol.

Tokens
9.2K
Snippets
19
Records
54
Agent score
83%

What's inside go-mysql-server

  1. Understand the query execution lifecycle

    main

    The go-mysql-server follows this execution flow:

    1. Parsing: The sql/parse package uses a vitess parser to translate a SQL query into a query plan.
    2. Analysis: The sql/analyzer takes the parsed plan and runs a series of rules (defined in rules.go) to resolve tables, columns, and databases, and to apply performance optimizations (such as index selection).
    3. Execution: The analyzed plan (composed of nodes from sql/plan and expressions from sql/expression) is executed recursively from the top of the tree to the bottom to retrieve results.
    4. Communication: The server package uses the MySQL wire protocol to send results back to the client.
  2. Implement sessions and transactions in a custom backend

    main

    Backends can manage session state and transactional semantics using the following patterns:

    Read-Only or Simple Backends

    For read-only implementations or those without complex state, you can re-use the sql.BaseSession object for sessioned access.

    Stateful Backends

    If your backend needs to store session-specific information (e.g., open data files that haven't been written yet), implement your own sql.Session. It is recommended to embed sql.BaseSession within your custom implementation to simplify the process.

    Transactional Backends

    To support transactional semantics, your session object must implement sql.TransactionSession and provide a corresponding sql.Transaction implementation. Note that implementation details for transactions are highly specific to the underlying data source.

  3. Implement a custom index driver

    main

    Index drivers allow for storing and querying indexes independently of the main table storage. To implement a custom driver, you must implement the following:

    1. sql.IndexDriver: The main driver interface. The ID() method must return a unique ID for the driver to prevent clashes. The driver is responsible for fault tolerance and recovering from index corruption.
    2. sql.Index: Returned by your driver when an index is created or loaded.
    3. sql.IndexValueIter: Returned by your sql.IndexLookup to provide the actual index values.

    Registration and Usage

    Register your driver in the sql.Context using:

    context.RegisterIndexDriver(mydriver)

    Once registered, you can create indexes using the USING driverid syntax in SQL:

    CREATE INDEX foo ON table USING driverid (col1, col2)
  4. Regenerate plan tests using Plangen

    main

    Plangen is a tool used to regenerate plan tests for existing plan suites. It is designed to help developers update plan suites when the underlying transform logic changes.

    Supported plan suites include:

    • PlanTests
    • IndexPlanTests
    • IntegrationPlanTests
    • ImdbPlanTests
    • TpchPlanTests
    • TpcdsPlanTests

    Important: Developers must manually verify the correctness of the plans generated by this tool to ensure they accurately reflect the intended execution logic.

  5. Test your backend implementation

    main

    To ensure your backend implementation is correct, use the following testing strategies:

    1. Engine Tests: Use the enginetest package to run the standard suite of engine tests provided by go-mysql-server. This validates that your implementation adheres to the expected engine behavior.
    2. Backend-Specific Tests: It is highly encouraged to write custom engine tests specific to your backend. This is critical when implementing complex features like transactions, which may not be covered by the standard in-memory test suite.
  6. Install go-mysql-server

    main

    Add go-mysql-server as a dependency to your Go project using go get.

    Regex Implementation Options

    1. ICU-compatible regex (Recommended for MySQL compatibility): Requires go-icu-regex, which has a Cgo dependency on ICU4C. You must have a C/C++ toolchain and libicu-dev (or equivalent) installed on your system.

    2. Pure Go regex (Non-compatible): Uses the Go standard library regex.Regex. This is not recommended for users seeking full MySQL compatibility and some tests may fail. To use this, compile with the -tags=gms_pure_go flag.

    go get github.com/dolthub/go-mysql-server@latest
  7. Implement native indexes for tables

    main

    To allow the engine to use native indexes for faster query execution, implement the sql.IndexedTable interface on your table objects.

    Your implementation must:

    1. Declare which indexes the table supports.
    2. Provide a mechanism for returning a subset of rows via an index.

    The sql.Index implementation's responsibility is to accept or reject combinations of sql.Range expressions. The engine uses these to construct a sql.IndexLookup struct, which is then passed to your sql.IndexedTable implementation.

  8. Use the in-memory test server

    main

    The built-in memory database implementation can be used as a stand-in for a real MySQL server in Go test environments. You can start the server by configuring a server.Config and initializing the engine and session using the memory package.

    Once the server is running, you can connect to it using any MySQL-compatible client (e.g., the mysql shell or Go MySQL connectors).

    package main
    
    import (
    	"context"
    	"fmt"
    	"time"
    
    	"github.com/dolthub/vitess/go/vt/proto/query"
    
    	sqle "github.com/dolthub/go-mysql-server"
    	"github.com/dolthub/go-mysql-server/memory"
    	"github.com/dolthub/go-mysql-server/server"
    	"github.com/dolthub/go-mysql-server/sql"
    	"github.com/dolthub/go-mysql-server/sql/types"
    )
    
    var (
    	dbName    = "mydb"
    	tableName = "mytable"
    	address   = "localhost"
    	port      = 3306
    )
    
    func main() {
    	pro := createTestDatabase()
    	engine := sqle.NewDefault(pro)
    
    	session := memory.NewSession(sql.NewBaseSession(), pro)
    	ctx := sql.NewContext(context.Background(), sql.WithSession(session))
    	ctx.SetCurrentDatabase(dbName)
    
    	config := server.Config{
    		Protocol: "tcp",
    		Address:  fmt.Sprintf("%s:%d", address, port),
    	}
    	s, err := server.NewServer(config, engine, sql.NewContext, memory.NewSessionBuilder(pro), nil)
    	if err != nil {
    		panic(err)
    	}
    	if err = s.Start(); err != nil {
    		panic(err)
    	}
    }
    
    func createTestDatabase() *memory.DbProvider {
    	db := memory.NewDatabase(dbName)
    	pro := memory.NewDBProvider(db)
    	session := memory.NewSession(sql.NewBaseSession(), pro)
    	ctx := sql.NewContext(context.Background(), sql.WithSession(session))
    
    	table := memory.NewTable(ctx, db, tableName, sql.NewPrimaryKeySchema(sql.Schema{
    		{Name: "name", Type: types.Text, Nullable: false, Source: tableName, PrimaryKey: true},
    		{Name: "email", Type: types.Text, Nullable: false, Source: tableName, PrimaryKey: true},
    		{Name: "phone_numbers", Type: types.JSON, Nullable: false, Source: tableName},
    		{Name: "created_at", Type: types.MustCreateDatetimeType(query.Type_DATETIME, 6), Nullable: false, Source: tableName},
    	}), db.GetForeignKeyCollection())
    	db.AddTable(tableName, table)
    
    	creationTime := time.Unix(0, 1667304000000001000).UTC()
    	_ = table.Insert(ctx, sql.NewRow("Jane Deo", "janedeo@gmail.com", types.MustJSON(`["556-565-566","777-777-777"]`), creationTime))
    	_ = table.Insert(ctx, sql.NewRow("Jane Doe", "jane@doe.com", types.MustJSON(`[]`), creationTime))
    	_ = table.Insert(ctx, sql.NewRow("John Doe", "john@doe.com", types.MustJSON(`["555-555-555"]`), creationTime))
    	_ = table.Insert(ctx, sql.NewRow("John Doe", "johnalt@doe.com", types.MustJSON(`[]`), creationTime))
    
    	return pro
    }