ksql

repository·master·Indexed 18 days ago

https://github.com/vingarcia/ksql

A simplified SQL interaction library for Golang providing a clean API for common database operations such as Insert, Patch, Delete, and Querying. KSQL is decoupled from specific database drivers via adapters, supporting Postgres (pgx v4/v5), MySQL, SQLServer, and SQLite. It includes features like generic data mapping into structs, transaction management, and an experimental query builder called kbuilder.

Tokens
18.3K
Snippets
63
Records
76
Agent score
61%

What's inside ksql

  1. Overview of KSQL

    master

    KSQL is a Go library designed for simple and satisfactory interaction with SQL databases. It provides a well-planned API to make learning, debugging, and avoiding common pitfalls easier.

    Key characteristics:

    • Decoupled Backend: KSQL is decoupled from its backend, allowing it to work on top of trusted technologies like pgx and database/sql via adapters.
    • Error Handling: Every operation returns errors a single time, simplifying error management.
    • Helper Functions: Includes built-in support for common operations like Insert, Patch, and Delete.
    • Data Mapping: Provides generic and powerful functions for querying and scanning data into structs, supporting sql.Scanner, sql.Valuer, and pgx special types (when using kpgx).
  2. Use KSQL Modifiers in struct tags

    master

    KSQL allows you to define special behaviors for struct fields using the ksql tag. These modifiers automate common tasks during database interactions.

    Supported Modifiers:

    • json: Saves the field as a JSON object in the database (e.g., ksql:"address,json"). Note that for SQLite, this is typically stored as a BLOB.
    • timeNowUTC: Automatically sets the field to time.Now().UTC() before saving.
    • timeNowUTC/skipUpdates: Sets the field to time.Now().UTC() during the initial creation but ignores it during subsequent updates (useful for created_at fields).

    Example struct definition:

    type User struct {
        ID        int       `ksql:"id"`
        Address   Address   `ksql:"address,json"` 
        UpdatedAt time.Time `ksql:"updated_at,timeNowUTC"` 
        CreatedAt time.Time `ksql:"created_at,timeNowUTC/skipUpdates"` 
    }
  3. Install KSQL database adapters

    master

    KSQL uses adapters to communicate with different databases. Choose the adapter that matches your database and driver requirements:

    • Postgres (pgx v4): github.com/vingarcia/ksql/adapters/kpgx
    • Postgres (pgx v5): github.com/vingarcia/ksql/adapters/kpgx5
    • MySQL: github.com/vingarcia/ksql/adapters/kmysql
    • SQLServer: github.com/vingarcia/ksql/adapters/ksqlserver
    • SQLite3 (CGO required): github.com/vingarcia/ksql/adapters/ksqlite3
    • SQLite (No CGO required): github.com/vingarcia/ksql/adapters/modernc-ksqlite
    go get github.com/vingarcia/ksql/adapters/kpgx
    go get github.com/vingarcia/ksql/adapters/kpgx5
    go get github.com/vingarcia/ksql/adapters/kmysql
    go get github.com/vingarcia/ksql/adapters/ksqlserver
    go get github.com/vingarcia/ksql/adapters/ksqlite3
    go get github.com/vingarcia/ksql/adapters/modernc-ksqlite
  4. Perform CRUD operations with KSQL

    master

    KSQL provides a high-level API for common database operations. You can use Insert, Delete, QueryOne, Query, and Patch to manage records.

    Common Operations:

    • Insert: Use db.Insert(ctx, table, &struct) to add new records. You can also insert inline.
    • Delete: Use db.Delete(ctx, table, id) to remove a record by its primary key.
    • Query One: Use db.QueryOne(ctx, &struct, "FROM table WHERE ...", args...) to retrieve a single record. If you omit the SELECT part, KSQL automatically builds it based on the struct fields.
    • Query Many: Use db.Query(ctx, &slice, "FROM table LIMIT ...") to retrieve multiple records into a slice. Note: Always use a LIMIT to avoid memory issues; for very large datasets, use QueryChunks.
    • Update (Patch): Use db.Patch(ctx, table, data) to update records. You can perform partial updates using two techniques:
      1. Anonymous Struct: Pass a struct containing only the ID and the fields you wish to change.
      2. Pointer/Nullable Fields: Use a struct where fields are pointers (or use nullable.Int, etc.). If a pointer is nil, that field will not be updated in the database.
    // Inserting
    err = db.Insert(ctx, UsersTable, &alison)
    
    // Deleting
    err = db.Delete(ctx, UsersTable, alison.ID)
    
    // Querying one
    var cris User
    err = db.QueryOne(ctx, &cris, "FROM users WHERE name = ?", "Cristina")
    
    // Querying many
    var users []User
    err = db.Query(ctx, &users, "FROM users LIMIT 10")
    
    // Partial Update (Technique 1: Anonymous Struct)
    err = db.Patch(ctx, UsersTable, struct {
        ID  int `ksql:"id"`
        Age int `ksql:"age"`
    }{ID: cris.ID, Age: 28})
    
    // Partial Update (Technique 2: Pointers/Nullable)
    err = db.Patch(ctx, UsersTable, PartialUpdateUser{
        ID:  cris.ID,
        Age: nullable.Int(28),
    })
  5. Execute database transactions with KSQL

    master

    KSQL supports transactions via the db.Transaction method. You provide a function that receives a ksql.Provider.

    • If the function returns an error, KSQL automatically performs a rollback.
    • If the function returns nil, KSQL commits the transaction.

    Warning: If you panic inside the transaction function, KSQL will not catch it to perform a rollback; it will re-panic to avoid hiding errors from the caller. Ensure you return errors for expected failure paths.

    err = db.Transaction(ctx, func(db ksql.Provider) error {
        var cris2 User
        err := db.QueryOne(ctx, &cris2, "FROM users WHERE id = ?", cris.ID)
        if err != nil {
            return err // Automatic rollback
        }
    
        err = db.Patch(ctx, UsersTable, PartialUpdateUser{
            ID:  cris2.ID,
            Age: nullable.Int(29),
        })
        if err != nil {
            return err // Automatic rollback
        }
    
        return nil // Commit
    })
  6. Querying data with KSQL

    master

    KSQL allows for flexible querying. You can query specific attributes into custom structs or load entire entities by omitting the SELECT part of the query if you provide a ksql.Table definition.

    Placeholder Syntax by Database:

    • Postgres: Use $1, $2, etc.
    • MySQL/SQLite: Use ?.
    • SQLServer: Use @p1, @p2, etc.

    Example: Querying into a custom struct

    var count []struct {
    	Count int    `ksql:"count"`
    	Type  string `ksql:"type"`
    }
    err = db.Query(ctx, &count, "SELECT type, count(*) as count FROM users GROUP BY type")

    Example: Loading entities (omitting SELECT)

    var adminUsers []User
    err = db.Query(ctx, &adminUsers, "FROM users WHERE type = $1", "admin")

    Example: Joining tables Use the tablename tag in your result struct to map joined rows to specific entities.

    var rows []struct {
    	OneUser    User    `tablename:"users"`
    	OneAddress Address `tablename:"addr"`
    }
    err = db.Query(ctx, &rows, `FROM users JOIN addresses addr ON users.user_id = addr.user_id`)
    // Querying custom attributes
    var count []struct {
    	Count int    `ksql:"count"`
    	Type  string `ksql:"type"`
    }
    err = db.Query(ctx, &count, "SELECT type, count(*) as count FROM users GROUP BY type")
    
    // Loading entities by omitting SELECT
    var adminUsers []User
    err = db.Query(ctx, &adminUsers, "FROM users WHERE type = $1", "admin")
    
    // Joining tables
    var rows []struct {
    	OneUser    User    `tablename:"users"`
    	OneAddress Address `tablename:"addr"`
    }
    err = db.Query(ctx, &rows, `FROM users JOIN addresses addr ON users.user_id = addr.user_id`)
  7. Set up the development environment for testing

    master

    To run the KSQL test suite, which uses docker-test to manage database instances, follow these steps:

    1. Install Docker: Ensure Docker is installed on your system.
    2. Configure Docker Permissions: Ensure you can run Docker without sudo. On Linux, add your user to the docker group:
      sudo usermod <your_username> -aG docker
      (Note: You must restart your session or reboot for this to take effect.)
    3. Pre-download Images: Run the following command once to prevent test timeouts during image downloads:
      make pre-download-all-images
    4. Run Tests: Execute the test suite using:
      make test
    # Add user to docker group
    sudo usermod <your_username> -aG docker
    
    # Download images
    make pre-download-all-images
    
    # Run tests
    make test
  8. How Query and WhereQueries work together

    master

    The kbuilder library uses a composition pattern to build complex SQL queries.

    1. WhereQuery: Represents a single atomic condition (e.g., age > %s) and its associated parameters.
    2. WhereQueries: A collection of WhereQuery objects. When built, it joins all conditions using the AND operator.
    3. Query: The top-level orchestrator. It takes the WhereQueries and integrates them into the full SELECT statement lifecycle.

    When Query.Build() is called, it iterates through the WhereQueries, resolves the driver-specific placeholders for each %s directive, and flattens all parameters into a single slice in the order they appear in the query.

  9. Use Mock to mock ksql.Provider for testing

    master

    The Mock struct implements the ksql.Provider interface, allowing you to simulate database behavior in unit tests.

    To mock a specific method, overwrite its corresponding Fn attribute (e.g., InsertFn for the Insert method).

    Best Practices:

    • Instantiate locally: Create a new Mock instance inside each unit test rather than using a global variable.
    • Capture values, don't assert inside mocks: Instead of performing assertions inside the mocked function, use closures to capture input values and assert against them in the test's assertion stage. This follows the (1) setup, (2) run, (3) assert pattern.
    • Handling multiple calls: To track multiple calls, use a closure to append captured values to a slice.
    // Capturing a single value
    var insertRecord interface{}
    dbMock := ksql.Mock{
    	InsertFn: func(ctx context.Context, table ksql.Table, record interface{}) error {
    		insertRecord = record
    	},
    }
    
    // Capturing multiple calls
    var insertRecords []interface{}
    dbMock := ksql.Mock{
    	InsertFn: func(ctx context.Context, table ksql.Table, record interface{}) error {
    		insertRecords = append(insertRecords, record)
    	},
    }
  10. How KSQL handles nested structs in queries

    master

    KSQL supports mapping database rows to nested Go structs.

    When using nested structs:

    1. Automatic SELECT generation: If you use a FROM clause, KSQL automatically builds the SELECT part of the query by traversing the nested struct fields and using their ksql tags.
    2. Constraint: If you manually provide a SELECT clause in your query string, KSQL will return an error because it cannot safely merge your manual selection with the automated nested field selection.
    3. Mapping: Fields are mapped using a prefix.column_name pattern (e.g., user.name).
  11. Inject a logger into a context to track queries

    master

    You can use ksql.InjectLogger to enable query logging for all subsequent database operations performed with a specific context.Context. This is a debugging tool that forces KSQL to log the query string, the parameters used, and any resulting errors whenever a query is executed.

    To use it, pass your context and a LoggerFn (such as the built-in ksql.Logger or ksql.ErrorLogger) to InjectLogger. The function returns a new context containing the logger, which you must then use for your database calls.

    // After injecting a logger into `ctx` all subsequent queries
    // that use this context will be logged.
    ctx = ksql.InjectLogger(ctx, ksql.Logger)
    
    // All the calls below will cause KSQL to log the queries:
    var user User
    db.Insert(ctx, usersTable, &user)
    
    user.Name = "NewName"
    db.Patch(ctx, usersTable, &user)
    
    var users []User
    db.Query(ctx, &users, someQuery, someParams...)
    db.QueryOne(ctx, &user, someQuery, someParams...)
    
    db.Delete(ctx, usersTable, user.ID)