repository·main·Indexed 43 days ago
https://github.com/duckdb/duckdb-goA Go SQL driver for DuckDB implementing the standard database/sql interface. It supports in-memory and persistent databases for analytical processing. Features include a high-performance Appender API for bulk data ingestion, a QueryAppender for complex bulk operations, and an opt-in Apache Arrow interface. The driver supports static and dynamic linking against DuckDB libraries and provides tools for managing connection lifetimes, profiling, and timestamp handling.
The duckdb-go driver relies on duckdb-go-bindings. The pre-built libraries statically link the following default extensions:
ICUJSONParquetAutocompleteAutomatic extension loading is enabled by default.
In the DuckDB C API, both TIMESTAMP and TIMESTAMP_TZ are stored as duckdb_timestamp (microseconds since 1970-01-01 UTC). When passing a Go time.Time to duckdb-go, it is converted to an instant using UnixMicro(), even if the target type is TIMESTAMP_TZ.
To force a specific DuckDB logical type for a query parameter (e.g., when DuckDB cannot infer the type or when you need high precision like TIMESTAMP_NS), use duckdb.Typed(value, typ).
Note: Bare time.Time values bind as TIMESTAMP_TZ by default.
start := time.Date(2024, time.April, 5, 0, 0, 0, 0, time.UTC)
end := time.Date(2024, time.April, 6, 0, 0, 0, 0, time.UTC)
row := db.QueryRow(`
SELECT COUNT(*)
FROM (VALUES (TIMESTAMP_NS '2024-04-05 12:00:00.000000001')) events_ns(ts)
WHERE ts >= ? AND ts < ?
`, duckdb.Typed(start, duckdb.TYPE_TIMESTAMP_NS), duckdb.Typed(end, duckdb.TYPE_TIMESTAMP_NS))Temporary objects (like temporary tables) are scoped to specific connections. Because Go's database/sql uses connection pooling, it may keep connections idle instead of closing them, causing temporary tables to persist longer than expected.
To ensure temporary objects are cleaned up immediately by preventing idle connection reuse, use db.SetMaxIdleConns(0).
To debug DuckDB's C++ source code from a Go application using duckdb-go, you must link your Go binary against a locally built DuckDB debug library and ensure the dynamic loader can locate the C++ sources via the correct rpath.
From the DuckDB repository root, run:
make debugUse the provided helper script scripts/go-build-duckdb-debug.sh to build your Go application. This script automatically exports CGO_CFLAGS and CGO_LDFLAGS (including -Wl,-rpath,...) and invokes go build with -tags duckdb_use_lib -gcflags "all=-N -l" to allow stepping through code.
Script Usage:
scripts/go-build-duckdb-debug.sh /absolute/path/to/your/go/app [package_or_dir] [output_name]
Example:
# Build a debug build from the DuckDB repo root:
make debug
# Build the Go application, linking to this DuckDB repo's debug library (`libduckdb`)
scripts/go-build-duckdb-debug.sh ~/Development/example-duckdb-go-project ./examples/simple simple_debugIf the loader cannot find libduckdb at runtime, set the following environment variable to the build/debug/src directory of the DuckDB repository:
DYLD_LIBRARY_PATHLD_LIBRARY_PATHTo use dynamic linking instead of the default static linking, pass the -tags=duckdb_use_lib flag to go build. You must ensure the libduckdb shared library (.so on Linux, .dylib on macOS) is available in your system path or specified via CGO_LDFLAGS and runtime environment variables like LD_LIBRARY_PATH or DYLD_LIBRARY_PATH.
# On Linux
CGO_ENABLED=1 CGO_LDFLAGS="-lduckdb -L/path/to/libs" go build -tags=duckdb_use_lib main.go
LD_LIBRARY_PATH=/path/to/libs ./main
# On MacOS
CGO_ENABLED=1 CGO_LDFLAGS="-lduckdb -L/path/to/libs" go build -tags=duckdb_use_lib main.go
DYLD_LIBRARY_PATH=/path/to/libs ./mainDuckDB runs in-process, meaning all its memory is allocated within your Go application's host process. For long-running applications, you must call the corresponding Close() functions to prevent memory leaks and ensure DuckDB synchronizes the Write-Ahead Log (WAL) to persistent storage.
Key resources to close include:
sql.DB (via db.Close())sql.Conn (via conn.Close())sql.Rows (via rows.Close())duckdb.Appender (via appender.Close())duckdb.Connector (via c.Close())duckdb.Arrow (via arrow.Release() on the reader)db, err := sql.Open("duckdb", "")
defer db.Close()
conn, err := db.Conn(context.Background())
defer conn.Close()
rows, err := conn.QueryContext(context.Background(), "SELECT 42")
rows.Close()
appender, err := duckdb.NewAppenderFromConn(conn, "", "test")
defer appender.Close()
c, err := duckdb.NewConnector("", nil)
defer c.Close()If you are using the older github.com/marcboeker/go-duckdb repository, you must migrate to github.com/duckdb/duckdb-go/v2 for versions v2.5.0 and later. Follow these steps:
gofmt.mapping or arrowmapping submodules, rewrite those paths as well.Note: All versions prior to v2.5.0 use the old import paths.
# Update dependency
go get github.com/duckdb/duckdb-go/v2@v2.5.0
# Rewrite import paths
gofmt -w -r '"github.com/marcboeker/go-duckdb/v2" -> "github.com/duckdb/duckdb-go/v2"' .
# If you use the mapping or arrowmapping submodules, also run
gofmt -w -r '"github.com/marcboeker/go-duckdb/mapping" -> "github.com/duckdb/duckdb-go/v2/mapping"' .
gofmt -w -r '"github.com/marcboeker/go-duckdb/arrowmapping" -> "github.com/duckdb/duckdb-go/v2/arrowmapping"' .
# Clean up
go mod tidyIf the bundled libraries do not meet your requirements, you can link against a custom static library (e.g., built from a local DuckDB checkout). You must enable CGO_ENABLED=1, set CPPFLAGS to define DUCKDB_STATIC_BUILD, and provide the correct CGO_LDFLAGS pointing to your library. Use the -tags=duckdb_use_static_lib flag during build or test.
# Example for Darwin ARM64
CGO_ENABLED=1 CPPFLAGS="-DDUCKDB_STATIC_BUILD" CGO_LDFLAGS="-lduckdb_bundle -lc++ -L/path/to/libs" go build -tags=duckdb_use_static_lib
# Example for testing against a local DuckDB checkout
CGO_ENABLED=1 \
CPPFLAGS="-DDUCKDB_STATIC_BUILD" \
CGO_LDFLAGS="-lduckdb_bundle -lc++ -L/path/to/duckdb/build/release" \
go test -tags=duckdb_use_static_lib ./...You can configure CLion to step from your Go application into DuckDB C++ code by importing two specific build configurations.
build-go-debug configuration: This is a shell configuration that uses scripts/go-build-duckdb-debug.sh. Replace the placeholder path ./../example-duckdb-go-project with your actual Go project path.go-debug configuration: This is a CLion External Run Configuration that depends on the build-go-debug task. Replace the placeholder path with your Go project path.go-debug configuration in debug mode.Note: Ensure you replace the script path in the build-go-debug configuration with the absolute path to your local copy of scripts/go-build-duckdb-debug.sh.
<!-- Configuration 1: build-go-debug -->
<component name="ProjectRunConfigurationManager">
<configuration default="false" name="build-go-debug" type="ShConfigurationType">
<option name="SCRIPT_TEXT" value="" />
<option name="INDEPENDENT_SCRIPT_PATH" value="true" />
<option name="SCRIPT_PATH" value="$PROJECT_DIR$/scripts/go-build-duckdb-debug.sh" />
<option name="SCRIPT_OPTIONS" value="$PROJECT_DIR$/../../example-duckdb-go-project $PROJECT_DIR$/../../example-duckdb-go-project/ go_debug" />
<option name="INDEPENDENT_SCRIPT_WORKING_DIRECTORY" value="true" />
<option name="SCRIPT_WORKING_DIRECTORY" value="$PROJECT_DIR$" />
<option name="INDEPENDENT_INTERPRETER_PATH" value="true" />
<option name="INTERPRETER_PATH" value="/bin/zsh" />
<option name="INTERPRETER_OPTIONS" value="" />
<option name="EXECUTE_IN_TERMINAL" value="true" />
<option name="EXECUTE_SCRIPT_FILE" value="true" />
<envs />
<method v="2" />
</configuration>
</component>
<!-- Configuration 2: go-debug -->
<component name="ProjectRunConfigurationManager">
<configuration default="false" name="go-debug" type="CLionExternalRunConfiguration" factoryName="Application" REDIRECT_INPUT="false" ELEVATE="false" USE_EXTERNAL_CONSOLE="false" EMULATE_TERMINAL="false" PASS_PARENT_ENVS_2="true" PROJECT_NAME="DuckDB" TARGET_NAME="duckdb" CONFIG_NAME="duckdb" RUN_PATH="$PROJECT_DIR$/../../example-duckdb-go-project/go_debug">
<method v="2">
<option name="CLION.EXTERNAL.BUILD" enabled="true" />
<option name="RunConfigurationTask" enabled="true" run_configuration_name="build-go-debug" run_configuration_type="ShConfigurationType" />
</method>
</configuration>
</component>To install the DuckDB Go driver, use the go get command. The driver conforms to the standard Go database/sql interface.
go get github.com/duckdb/duckdb-go/v2The DuckDB Arrow Interface is a heavy dependency and is now opt-in. To use it, you must pass the -tags=duckdb_arrow flag during your Go build process.
go build -tags=duckdb_arrowThe ScalarBinder allows you to initialize or update a context.Context during the binding phase of the UDF execution. This is useful for passing arbitrary data or state from the binding phase into the execution phase.
ScalarBinder is a function of type ScalarBinderFn: func(parentCtx context.Context, args []ScalarUDFArg) (context.Context, error).
Arguments are provided via ScalarUDFArg, which contains:
Foldable: Boolean indicating if the argument was folded into a constant value.Value: The driver.Value of the argument (if foldable).If you provide a ScalarBinder, the resulting context will be passed to your RowContextExecutor or ChunkContextExecutor.