chdb-go

repository·main·Indexed 40 days ago

https://github.com/chdb-io/chdb-go

Go bindings and a CLI for chDB, a high-performance analytical database engine. It supports system-installed libraries, embedded binaries for self-contained distribution, and a standard database/sql driver. Features include stateless queries, stateful sessions, and durable storage in object stores like S3, R2, and MinIO via the chdb/durable package.

Tokens
9.4K
Snippets
29
Records
42
Agent score
49%

What's inside chdb-go

  1. Understand how chdb-go locates the libchdb engine

    main

    At runtime, chdb-go searches for libchdb using the following priority order:

    1. CHDB_LIB_PATH: If this environment variable is set, it points directly to the library file. This disables all other search methods. If the file at this path is invalid, it results in an error.
    2. Embedded Engine: If the build imports an engine module (e.g., lib/embedded), the engine inside the binary is used. No further searching occurs.
    3. Executable Directory: The directory containing the running executable.
    4. PATH: Standard system path search.
    5. System Library Paths: /usr/local/lib, /opt/homebrew/lib, and /usr/lib.
    6. Dynamic Loader Paths: Directories in LD_LIBRARY_PATH, DYLD_LIBRARY_PATH, and DYLD_FALLBACK_LIBRARY_PATH.
    7. Dynamic Loader: Search by name (e.g., ldconfig-registered installs or RUNPATH).

    Tip: Use chdbpurego.LoadedLibraryPath() to log the absolute path of the library actually being used by your process.

  2. Interpret chdb-go version strings

    main

    The versioning follows a specific format: v[ClickHouse Line][chdb-core counter].[packaging count].

    Example: v0.260700.1

    • 26 07 00: Represents chdb-core v26.7.0 (on the ClickHouse 26.7 line).
    • 1: The first packaging of this specific engine.

    To check versions at runtime:

    • Use chdbpurego.EmbeddedEngineVersion() to get the chdb-core release.
    • Run SELECT version() within chdb to get the exact ClickHouse build.
  3. Implement durable storage with chdb/durable

    main

    The chdb/durable package allows you to store a chDB database's authoritative state in object storage (S3, R2, MinIO, or a local directory). It uses a checkpoint + write-ahead-log (WAL) protocol.

    Key Concepts:

    • Namespace: Represents the storage location (e.g., s3://my-bucket/path).
    • Object: Represents a specific tenant/database within that namespace.
    • Durability: Achieved via FlushThrough(ctx, ticket) (for a specific statement) or Checkpoint(ctx) (to fold WAL into a new base).
    • Execution: Execute runs the statement locally and joins the WAL buffer. It does not mean the data has left the machine yet.
    • Constraints: One process can only hold one open durable object at a time. Access control is managed via your object store's IAM.

    Error Handling:

    • durable.ErrLeaseHeld: Another writer is currently active.
    • Use durable.CategoryOf(err) to distinguish between CategoryCommitAmbiguous and CategoryCorrupt.
    import "github.com/chdb-io/chdb-go/v2/chdb/durable"
    
    // Setup namespace
    ns, err := durable.NewNamespace("s3://my-bucket/durable?region=eu-west-1",
            durable.NamespaceOptions{Owner: "worker-1"})
    
    // Open an object
    obj, existed, err := ns.Open(ctx, "tenant-123", durable.OpenOptions{Database: "mem"})
    defer obj.Close(ctx)
    
    if !existed {
            _, err = obj.Execute(ctx, "CREATE TABLE events (id UInt64) ENGINE = MergeTree ORDER BY id")
    }
    
    // Execute and ensure durability
    ticket, err := obj.Execute(ctx, "INSERT INTO events VALUES (1)")
    err = obj.FlushThrough(ctx, ticket) // survives machine loss
    
    // Query and Checkpoint
    rows, err := obj.Query(ctx, "SELECT count() FROM events", "JSONEachRow")
    _, err = obj.Checkpoint(ctx) // folds base + WAL into a fresh base
  4. Understand the EmbeddedEngine abstraction

    main

    The EmbeddedEngine struct describes a libchdb payload that is compiled directly into the calling Go binary. This allows for a 'zero-install' experience where go get is sufficient to run chDB.

    • Version: The chdb-core release version used for diagnostics.
    • FileName: The name the library must be written under (e.g., libchdb.so on macOS).
    • Digest: The lowercase hex SHA-256 of the extracted library, used for safe concurrent extraction.
    • Size: The extracted size in bytes.
    • Open() (io.ReadCloser, error): Returns the extracted library bytes.
  5. Analyze queries with QueryAnalysis and QueryClass

    main

    The QueryAnalysis type provides metadata about a statement, allowing you to classify its impact on the system state. This is useful for security gating or permission management.

    QueryClass levels (from least to most restrictive):

    • QueryReadOnly (0): SELECT, SHOW, DESCRIBE, EXPLAIN. Leaves no trace.
    • QueryMutating (1): INSERT, CREATE, ALTER, DROP. Changes the database.
    • QueryMutatingGlobal (2): Global UDFs, named collections, or writes to system. Persistent and outside standard database checkpoints.
    • QueryControl (3): USE, SET, SYSTEM, BACKUP, RESTORE, or writes outside the engine.
    • QueryUnknown (4): The statement failed to parse or is unclassifiable. Treat this as a refusal/unsafe.

    QueryAnalysis Fields:

    • StatementCount: Number of executable statements.
    • Class: The QueryClass of the statement.
    • HasSecrets: Boolean indicating if the text contains credentials (passwords, keys, etc.).
    • WritesOnlyTargetDatabase: True if all persistent writes land in the named target database.
    • ChangesDatabaseLifecycle: True if the statement creates, drops, or renames a database.
  6. Install chdb-go with an embedded engine for self-contained binaries

    main

    To create a self-contained binary (e.g., for FROM scratch Docker images or environments where you cannot perform a system-level install), use the embedded engine module. The engine is included in your binary and is extracted to a cache directory on the first run.

    Add a blank import to your Go code:

    import _ "github.com/chdb-io/chdb-go/lib/embedded"
  7. Manage the chDB engine lifecycle with Shutdown

    main

    To stop the engine and join its threads, call Shutdown().

    Important constraints:

    • Prerequisite: All connections must be closed and all results must be freed before calling Shutdown(). If connections are still open, Shutdown will return an error.
    • Terminal Operation: Shutdown is terminal. Once called, you cannot open new connections via NewConnection in the same process; you must restart the process to use chDB again.
    • Idempotency: Calling Shutdown more than once is harmless.
    • Safety: If the process exits normally, threads are reaped by the OS, but calling Shutdown is recommended for orderly stops, especially when using sanitizers or host runtimes with finalizers.
    import "github.com/chdb-io/chdb-go/v2/chdb-purego"
    
    // ... use connections ...
    
    err := chdbpurego.Shutdown()
    if err != nil {
        // handle error
    }
  8. Manage the engine lifecycle with Shutdown()

    main

    Call Shutdown once when your program is completely finished with chDB. This stops the engine and joins the background threads started by chDB.

    Key Behaviors:

    • Terminal Operation: Shutdown is terminal. After calling it, NewSession will fail for the remainder of the process. Do not call it between units of work.
    • Prerequisite: All Session instances must be closed before calling Shutdown. If sessions are still open, Shutdown will return an error wrapping ErrSessionsOpen.
    • No Engine Running: If no session was ever opened (e.g., you only called EngineVersion), Shutdown returns nil and does nothing.
    • Safety: If you don't call Shutdown, the threads will still be reaped by the OS when the process exits.
    // Correct lifecycle pattern
    sess, err := chdb.NewSession("/path/to/data")
    // ... use session ...
    sess.Close()
    
    err = chdb.Shutdown()
    if err != nil {
        // handle error
    }
  9. Install the chdb-go CLI

    main

    You can install the chdb-go command-line interface directly via Go:

    go install github.com/chdb-io/chdb-go/v2@latest

    Run the CLI using:

    $GOPATH/bin/chdb-go

    The CLI uses the same engine lookup logic as the Go library and requires an engine to be available on the machine (unless using an embedded build).

    go install github.com/chdb-io/chdb-go/v2@latest
    $GOPATH/bin/chdb-go
  10. Install chdb-go using a machine-installed engine

    main

    If you prefer to manage the libchdb engine separately from your Go application, install the library to your machine first. Note that upgrading chdb-go does not upgrade the engine; you must re-run the installer or update_libchdb.sh to keep them in sync.

    1. Install libchdb on your machine:
      curl -sL https://lib.chdb.io | bash
    2. Get the Go bindings:
      go get github.com/chdb-io/chdb-go/v2
    curl -sL https://lib.chdb.io | bash
    go get github.com/chdb-io/chdb-go/v2
  11. How to manually select a specific libchdb engine version

    main
    While the embedded package automatically selects the correct engine for your platform, you can bypass this automatic selection by importing a specific platform module directly. This is useful if you need to pin a specific engine version in your go.mod rather than relying on the version managed by the embedded module.
  12. Configure the embedded engine extraction directory

    main

    When using an embedded engine, the library is extracted to a cache directory. You can control this location using the CHDB_CACHE_DIR environment variable.

    • Default behavior: The user cache directory, falling back to the temporary directory.
    • Security requirement: The directory must be writable only by the user running the process to prevent substitution attacks.