duckdb-rs

repository·main·Indexed 21 days ago

https://github.com/duckdb/duckdb-rs

An ergonomic Rust wrapper for DuckDB providing type-safe bindings and an API inspired by rusqlite. It supports high-performance data integration with Arrow, Parquet, and Polars, and enables the development of custom DuckDB extensions in Rust. The crate offers multiple installation methods, including a bundled feature to compile DuckDB from source and support for connection pooling via r2d2.

Tokens
22.4K
Snippets
76
Records
88
Agent score
76%

What's inside duckdb-rs

  1. Thread safety and Connection pooling in duckdb-rs

    main

    A Connection in duckdb-rs is Send but not Sync. This means you can move a connection between threads, but you cannot share a single connection across multiple threads simultaneously.

    To handle multi-threaded applications, you should provide each thread with its own connection. The recommended way to do this is using a connection pool via the r2d2 feature.

    Example using r2d2:

    use duckdb::{DuckdbConnectionManager, params};
    
    let manager = DuckdbConnectionManager::file("file.db")?;
    let pool = r2d2::Pool::new(manager)?;
    
    // Each worker checks out its own connection from the pool.
    let conn = pool.get()?;
    conn.execute("INSERT INTO foo (bar) VALUES (?)", params![1])?;
    use duckdb::{DuckdbConnectionManager, params};
    
    let manager = DuckdbConnectionManager::file("file.db")?;
    let pool = r2d2::Pool::new(manager)?;
    
    let conn = pool.get()?;
    conn.execute("INSERT INTO foo (bar) VALUES (?)", params![1])?;
  2. How to handle the ICU extension with bundled features

    main

    When using the bundled feature, the ICU extension is omitted to comply with crates.io's 10MB package size limit. Without ICU, certain date/time operations (e.g., now() - interval '1 day') will fail.

    You have three ways to resolve this:

    1. Runtime Loading: Load the extension at runtime using SQL: conn.execute_batch("INSTALL icu; LOAD icu;")?
    2. System Library: Link against a system-installed libduckdb that was compiled with ICU support.
    3. Bundled-CMake: Use the bundled-cmake feature with the icu feature enabled (requires a git/workspace checkout).
    conn.execute_batch("INSTALL icu; LOAD icu;")?;
  3. Choosing between a Database Client and a DuckDB Extension

    main

    The duckdb-rs crate serves two distinct purposes. You should choose exactly one based on your goal:

    1. Database Client (Most Common): Use this to embed DuckDB in your Rust application to run queries. Do not enable the loadable-extension feature for this use case.
    2. DuckDB Extension: Use this to build a loadable .duckdb_extension that DuckDB loads at runtime. Enable the loadable-extension feature for this.

    Warning: If you enable loadable-extension in a regular client application, calls like Connection::open_in_memory() will panic because the DuckDB API is only initialized when the extension is loaded by DuckDB.

  4. Quickstart with duckdb-rs

    main

    To get started with duckdb-rs, create a new Rust project and add the duckdb crate with the bundled feature enabled. This allows you to use DuckDB without needing to install it on your system separately.

    1. Setup

    cargo new quack-in-rust
    cd quack-in-rust
    cargo add duckdb -F bundled

    2. Basic Usage

    You can open an in-memory connection, execute SQL commands, and map query results to Rust structs using Connection::open_in_memory() and query_map.

    use duckdb::{params, Connection, Result};
    
    struct Duck {
        id: i32,
        name: String,
    }
    
    fn main() -> Result<()> {
        let conn = Connection::open_in_memory()?;
    
        conn.execute(
            "CREATE TABLE ducks (id INTEGER PRIMARY KEY, name TEXT)",
            [], // empty list of parameters
        )?;
    
        conn.execute_batch(
            r"#
            INSERT INTO ducks (id, name) VALUES (1, 'Donald Duck');
            INSERT INTO ducks (id, name) VALUES (2, 'Scrooge McDuck');
            "#,
        )?;
    
        conn.execute(
            "INSERT INTO ducks (id, name) VALUES (?, ?)",
            params![3, "Darkwing Duck"],
        )?;
    
        let ducks = conn
            .prepare("FROM ducks")?
            .query_map([], |row| {
                Ok(Duck {
                    id: row.get(0)?,
                    name: row.get(1)?,
                }
            })?
            .collect::<Result<Vec<_>>>()?;
    
        for duck in ducks {
            println!("{}) {}", duck.id, duck.name);
        }
    
        Ok(())
    }
  5. Link against a system-installed DuckDB library

    main

    If you prefer not to use the bundled features, you can link against a DuckDB library already present on your system. You must provide the paths to the library and the header file using environment variables.

    Linux ARM64 Example:

    wget https://github.com/duckdb/duckdb/releases/download/v1.5.5/libduckdb-linux-arm64.zip
    unzip libduckdb-linux-arm64.zip -d libduckdb
    
    export DUCKDB_LIB_DIR=$PWD/libduckdb
    export DUCKDB_INCLUDE_DIR=$DUCKDB_LIB_DIR
    export LD_LIBRARY_PATH=$DUCKDB_LIB_DIR
    
    cargo build --examples

    macOS Example:

    wget https://github.com/duckdb/duckdb/releases/download/v1.5.5/libduckdb-osx-universal.zip
    unzip libduckdb-osx-universal.zip -d libduckdb
    
    export DUCKDB_LIB_DIR=$PWD/libduckdb
    export DUCKDB_INCLUDE_DIR=$DUCKDB_LIB_DIR
    export DYLD_FALLBACK_LIBRARY_PATH=$DUCKDB_LIB_DIR
    
    cargo build --examples
    export DUCKDB_LIB_DIR=$PWD/libduckdb
    export DUCKDB_INCLUDE_DIR=$DUCKDB_LIB_DIR
  6. Install duckdb-rs

    main

    Add the crate to your project using cargo add with the bundled feature to avoid manual DuckDB installation:

    cargo add duckdb -F bundled

    Or manually in Cargo.toml:

    [dependencies]
    duckdb = { version = "~1.10505.0", features = ["bundled"] }

    Using Git (Development Version)

    To use the latest development version from the main branch:

    duckdb = { git = "https://github.com/duckdb/duckdb-rs", branch = "main", features = ["bundled"] }

    To use the LTS release branch (DuckDB 1.4 Andium):

    duckdb = { git = "https://github.com/duckdb/duckdb-rs", branch = "v1.4-andium", features = ["bundled"] }

    Versioning Note

    duckdb-rs versions encode the DuckDB version in the second semver component (e.g., DuckDB v1.5.0 is 1.10500.x). Use a tilde requirement (~) to receive patch releases without automatically upgrading the bundled DuckDB version.

    cargo add duckdb -F bundled
  7. Automate DuckDB installation with DUCKDB_DOWNLOAD_LIB

    main

    Setting the DUCKDB_DOWNLOAD_LIB=1 environment variable instructs the build script to download pre-built DuckDB binaries from GitHub Releases. This automates the manual process of downloading and setting paths. The downloaded version matches the DuckDB version encoded in the libduckdb-sys crate.

    Note: Leave DUCKDB_STATIC unset, as the downloaded archives do not contain a static library.

    Usage:

    DUCKDB_DOWNLOAD_LIB=1 cargo test
    DUCKDB_DOWNLOAD_LIB=1 cargo test
  8. Install duckdb-rs using the bundled-cmake feature

    main

    The bundled-cmake feature is an experimental option that builds DuckDB from a local checkout using upstream CMake instead of the cc backend. This is useful for enabling CMake-only extensions like icu.

    Key behaviors:

    • It implies the bundled feature.
    • It automatically enables the parquet feature.
    • It enables upstream jemalloc on supported 64-bit, non-musl Linux targets. To force the standard allocator, set DUCKDB_DISABLE_JEMALLOC=1.
    • Extension autoload/autoinstall is enabled by default. To disable, set DUCKDB_DISABLE_EXTENSION_LOAD=1 or DISABLE_EXTENSION_LOAD=1.
    • It builds in Release mode by default. To override, use DUCKDB_CMAKE_BUILD_TYPE or CMAKE_BUILD_TYPE.
    • Requires a git/workspace checkout; published crates on crates.io do not include the full source tree required for this feature.

    Example usage:

    [dependencies]
    duckdb = { git = "https://github.com/duckdb/duckdb-rs", branch = "main", features = ["bundled-cmake", "icu"] }
    [dependencies]
    duckdb = { git = "https://github.com/duckdb/duckdb-rs", branch = "main", features = ["bundled-cmake", "icu"] }
  9. Install duckdb-rs using the bundled feature

    main

    The simplest way to install duckdb-rs and avoid build problems is to use the bundled feature. This tells libduckdb-sys to compile DuckDB from source using the cc crate and link against it. The source is embedded in the crate and tracks the DuckDB version vendored for that specific release.

    To enable this, add the following to your Cargo.toml:

    [dependencies]
    duckdb = { version = "~1.10505.0", features = ["bundled"] }
    cargo add duckdb --features bundled
  10. How to build a DuckDB extension in Rust

    main

    To build a valid DuckDB extension, you cannot simply run cargo build. DuckDB requires extensions to be files ending in .duckdb_extension that contain a specific metadata footer matching the target DuckDB version.

    To handle the metadata and platform/version detection correctly, it is highly recommended to use the official extension-template-rs template.

    Using the template, you can run make debug to build a shared library and automatically append the required footer, resulting in a valid .duckdb_extension file that can be loaded via duckdb -unsigned.

  11. Iterate over query results using Rows

    main

    The Rows struct is a lazy handle for the resulting rows of a query. It implements FallibleStreamingIterator, meaning each call to next() can return an error.

    Important: Rows is not compatible with Rust's standard Iterator trait because the lifetime of the returned Row is tied to the lifetime of the Rows handle. To avoid manual loop management, use Statement::query_map or Statement::query_and_then which return standard Iterator types.

    To iterate manually, use a while let loop with rows.next()?.

    use duckdb::{Connection, Result};
    
    fn process_results(conn: &Connection) -> Result<()> {
        let mut stmt = conn.prepare("SELECT id, name FROM people")?;
        let mut rows = stmt.query([])?;
    
        while let Some(row) = rows.next()? {
            // Process row...
        }
        Ok(())
    }
  12. Use the `Value` enum for owning DuckDB scalar values

    main

    The Value enum is an owning container for DuckDB's dynamic types. It is used to represent scalar values when interacting with the database. Note that the specific variant of Value is typically determined by DuckDB's logical types rather than the caller.

    For non-owning (borrowed) access to dynamic values, use ValueRef instead.