DOCSEARCHbyAlgolia
Submit Docs

DuckDB-Wasm Documentation

repository·main·Indexed 24 days ago

https://github.com/duckdb/duckdb-wasm

A WebAssembly port of the DuckDB OLAP database engine for the browser and Node.js. It enables complex SQL queries on Parquet, CSV, and JSON files in client-side environments. The project includes the core Wasm library, a TypeScript API (@duckdb/duckdb-wasm), a Rust-based SQL shell (@duckdb/duckdb-wasm-shell), and React hooks (@duckdb/react-duckdb). Key features include support for Arrow IPC streams, lazy chunked results for large datasets, and a flexible extension system for core and community plugins.

Tokens
10.1K
Snippets
15
Records
53
Agent score
84%

What's inside DuckDB-Wasm

  2. Understand the differences between DuckDB and DuckDB-Wasm

    main

    DuckDB-Wasm is a WebAssembly port of DuckDB designed to run in browsers and Node.js. While it is based on DuckDB (currently v1.5.4), there are key behavioral differences to note:

    • Network & HTTP: The default HTTP stack is different. LOAD httpfs in Wasm uses a JavaScript re-implementation. All requests are automatically upgraded to HTTPS, and servers must allow Cross-Origin (CORS) access.
    • Extension Loading: Extension installation is lazy. INSTALL extension_name FROM 'url'; defers fetching until the first LOAD extension_name; instruction. Shorthands like INSTALL x FROM community; are supported.
    • Bundling & Autoloading: Unlike native DuckDB where core extensions (JSON, Parquet, ICU, etc.) are often bundled, DuckDB-Wasm autoloads them at runtime via network fetches. Note that ICU autoloading may fail in some cases, requiring an explicit LOAD icu;.
    • Threading: DuckDB-Wasm is single-threaded by default. Multithreading is currently experimental.
    • Sandboxing: It is sandboxed and may have different support for out-of-core operations and direct filesystem access compared to the native version.

  3. Instantiate DuckDB-Wasm

    main

    To use DuckDB-Wasm, you must first instantiate an AsyncDuckDB instance. This process involves selecting an appropriate bundle (MVP or EH) based on browser capabilities, creating a Web Worker, and then calling .instantiate() on the AsyncDuckDB object.

    Depending on your build tool (jsDelivr CDN, Webpack, Vite, or static serving), the method for loading the .wasm modules and worker scripts will vary. Always ensure you use a ConsoleLogger to monitor the database lifecycle.

    import * as duckdb from '@duckdb/duckdb-wasm';

const JSDELIVR_BUNDLES = duckdb.getJsDelivrBundles();

// Select a bundle based on browser checks
const bundle = await duckdb.selectBundle(JSDELIVR_BUNDLES);

const worker_url = URL.createObjectURL(
  new Blob([`importScripts("${bundle.mainWorker!}");`], {type: 'text/javascript'})
);

// Instantiate the asynchronous version of DuckDB-wasm
const worker = new Worker(worker_url);
const logger = new duckdb.ConsoleLogger();
const db = new duckdb.AsyncDuckDB(logger, worker);
await db.instantiate(bundle.mainModule, bundle.pthreadWorker);
URL.revokeObjectURL(worker_url);

  4. Import data into DuckDB-Wasm

    main

    DuckDB-Wasm supports importing data from various formats including Arrow, CSV, JSON, Parquet, and standard JavaScript arrays.

    Common patterns include:

    • Arrow: Use insertArrowTable for existing arrow.Table objects or insertArrowFromIPCStream for raw IPC streams.
    • CSV/JSON: Register the file using registerFileText or registerFileBuffer, then use insertCSVFromPath or insertJSONFromPath on a connection. You can provide typed options like schema, delimiter, and columns for precise control.
    • Parquet: Use registerFileHandle (for local File objects), registerFileURL (for HTTP), or registerFileBuffer (for Uint8Array).
    • SQL-based: You can directly query remote files by specifying their URL in a SELECT statement.

    Note: Always call connection.close() to release memory after data operations are complete.

    // Data can be inserted from an existing arrow.Table
await c.insertArrowTable(existingTable, { name: 'arrow_table' });

// ..., from a raw Arrow IPC stream
const c = await db.connect();
const streamResponse = await fetch(`someapi`);
const streamReader = streamResponse.body.getReader();
const streamInserts = [];
while (true) {
    const { value, done } = await streamReader.read();
    if (done) break;
    streamInserts.push(c.insertArrowFromIPCStream(value, { name: 'streamed' }));
}
await Promise.all(streamInserts);

// ..., from CSV files
await db.registerFileText(`data.csv`, '1|foo\n2|bar\n');
await c.insertCSVFromPath('data.csv', {
    schema: 'main',
    name: 'foo',
    detect: false,
    header: false,
    delimiter: '|',
    columns: {
        col1: new arrow.Int32(),
        col2: new arrow.Utf8(),
    },
});

// ..., from Parquet files
const pickedFile: File = letUserPickFile();
await db.registerFileHandle('local.parquet', pickedFile, DuckDBDataProtocol.BROWSER_FILEREADER, true);
await db.registerFileURL('remote.parquet', 'https://origin/remote.parquet', DuckDBDataProtocol.HTTP, false);

// ..., by specifying URLs in the SQL text
await c.query(`
    CREATE TABLE direct AS
        SELECT * FROM "https://origin/remote.parquet"
`);

// Close the connection to release memory
await c.close();

  5. Install and load DuckDB extensions in DuckDB-Wasm

    main

    DuckDB-Wasm supports core, community, and external extensions. You can load them explicitly, via autoloading, or by registering them from specific repositories.

    Explicit Loading Use LOAD extension_name; to load an extension.

    Autoloading Many extensions (like parquet or json) are automatically loaded when you use relevant functions, such as read_parquet().

    Registering from Repositories You can install extensions from specific URLs or community shorthands:

    • INSTALL extension_name FROM community;
    • INSTALL extension_name FROM 'https://repository.endpoint.org';

    To verify which extensions are currently loaded, query the duckdb_extensions() function.

    --- Explicitly load extensions
LOAD icu;

--- Or have them autoloaded when using relevant functions or settings
DESCRIBE FROM read_parquet('https://blobs.duckdb.org/stations.parquet');

--- Or register extensions
INSTALL h3 FROM community;
INSTALL sqlite_scanner FROM 'https://extensions.duckdb.org';
INSTALL quack FROM 'https://community-extensions.duckdb.org';

--- And then load them
LOAD h3;
LOAD sqlite_scanner;
LOAD quack;

--- Check loaded extensions
SELECT * FROM duckdb_extensions() WHERE loaded;

  6. Build DuckDB-Wasm from source

    main

    To build the project from source, ensure you have the necessary build tools installed, then follow these steps to clone the repository, initialize submodules, apply patches, and start a local server.

    git clone https://github.com/duckdb/duckdb-wasm.git
cd duckdb-wasm
git submodule init
git submodule update
make apply_patches
make serve

  7. Use @duckdb/react-duckdb for React integration

    main

    The @duckdb/react-duckdb package provides React Context providers and hooks to integrate DuckDB-Wasm into React applications. It exports several key modules for managing the DuckDB lifecycle, database connections, and platform-specific configurations:

    • connection_provider: Manages and provides DuckDB connections via React Context.
    • database_provider: Manages the DuckDB database instance.
    • platform_provider: Handles platform-specific setup (e.g., WebWorker vs Node.js).
    • resolvable: Utilities for handling asynchronous or resolvable values within the React lifecycle.

  8. Implement or use the DuckDBRuntime interface

    main

    The DuckDBRuntime interface defines the low-level capabilities required to bridge the WASM module with the host environment (Browser or Node.js). It includes APIs for file management, directory operations, and UDF execution.

    Key functional groups include:

    File Management (via File ID)

    • openFile(mod, fileId, flags): Opens a file using its unique identifier.
    • syncFile(mod, fileId): Synchronizes the file.
    • closeFile(mod, fileId): Closes the file.
    • readFile(mod, fileId, buffer, bytes, location): Reads data from a file into a buffer.
    • writeFile(mod, fileId, buffer, bytes, location): Writes data from a buffer to a file.

    Directory and Path Operations

    • checkFile(mod, pathPtr, pathLen): Checks if a file exists at the given path.
    • createDirectory(mod, pathPtr, pathLen): Creates a directory.
    • listDirectoryEntries(mod, pathPtr, pathLen): Lists entries in a directory.
    • removeFile(mod, pathPtr, pathLen): Removes a file.

    Asynchronous File Handles

    For files that require asynchronous acquisition (like network-based files), the runtime provides:

    • prepareFileHandle(path, protocol): Returns a Promise<PreparedDBFileHandle[]>.
    export interface DuckDBRuntime {
    _files?: Map<string, any>;
    _udfFunctions: Map<number, UDFFunction>;

    testPlatformFeature(mod: DuckDBModule, feature: number): boolean;

    getDefaultDataProtocol(mod: DuckDBModule): number;
    openFile(mod: DuckDBModule, fileId: number, flags: FileFlags): void;
    syncFile(mod: DuckDBModule, fileId: number): void;
    closeFile(mod: DuckDBModule, fileId: number): void;
    dropFile(mod: DuckDBModule, fileNamePtr: number, fileNameLen: number): void;
    getLastFileModificationTime(mod: DuckDBModule, fileId: number): number;
    truncateFile(mod: DuckDBModule, fileId: number, newSize: number): void;
    readFile(mod: DuckDBModule, fileId: number, buffer: number, bytes: number, location: number): number;
    writeFile(mod: DuckDBModule, fileId: number, buffer: number, bytes: number, location: number): number;

    removeDirectory(mod: DuckDBModule, pathPtr: number, pathLen: number): void;
    checkDirectory(mod: DuckDBModule, pathPtr: number, pathLen: number): boolean;
    createDirectory(mod: DuckDBModule, pathPtr: number, pathLen: number): void;
    listDirectoryEntries(mod: DuckDBModule, pathPtr: number, pathLen: number): boolean;
    glob(mod: DuckDBModule, pathPtr: number, pathLen: number): void;
    moveFile(mod: DuckDBModule, fromPtr: number, fromLen: number, toPtr: number, toLen: number): void;
    checkFile(mod: DuckDBModule, pathPtr: number, pathLen: number): boolean;
    removeFile(mod: DuckDBModule, pathPtr: number, pathLen: number): void;

    prepareFileHandle?: (path: string, protocol: DuckDBDataProtocol) => Promise<PreparedDBFileHandle[]>;
    prepareFileHandles?: (path: string[], protocol: DuckDBDataProtocol) => Promise<PreparedDBFileHandle[]>;
    prepareDBFileHandle?: (path: string, protocol: DuckDBDataProtocol) => Promise<PreparedDBFileHandle[]>;

    progressUpdate(final: number, percentage: number, iteration: number): void;

    callScalarUDF(
        mod: DuckDBModule,
        response: number,
        funcId: number,
        descPtr: number,
        descSize: number,
        ptrsPtr: number,
        ptrsSize: number,
    ): void;
}

  9. Manage DuckDB-Wasm via WebDB and Connection

    main

    The WebDB class serves as the primary entry point for managing a DuckDB instance in a web environment, while the Connection class provides the interface for executing queries and managing data.

    To use DuckDB-Wasm, you typically create a WebDB instance (or retrieve the static instance via Get()), open it using Open(), and then create a Connection using Connect(). All query execution and data manipulation occur through the Connection object.

  10. Use Prepared Statements in DuckDB-Wasm

    main

    Prepared statements allow you to pre-compile a SQL query and execute it multiple times with different parameters, which is more efficient and secure.

    Workflow:

    1. Create a statement using conn.prepare(sql).
    2. Execute the statement using .query(params) for materialized results or .send(params) for chunked results.
    3. Explicitly call .close() on the statement to release resources. Note that closing the parent connection will also release all associated statements.
    // Prepare query
const stmt = await conn.prepare(`SELECT v + ? FROM generate_series(0, 10000) as t(v);`);

// ... and run the query with materialized results
await stmt.query(234);

// ... or result chunks
for await (const batch of await stmt.send(234)) {
    // ...
}

// Close the statement to release memory
await stmt.close();

// Closing the connection will release statements as well
await conn.close();

  11. Execute queries in DuckDB-Wasm

    main

    You can execute SQL queries using a connection object in two ways:

    1. Materialized Results: Use conn.query<T>(sql) to fetch the entire result set at once. This is useful for smaller datasets where you want to work with the full result immediately.
    2. Lazy Chunked Results: Use conn.send<T>(sql) to iterate over result chunks using an async iterator. This is more memory-efficient for large datasets as it processes data in batches.
    // Either materialize the query result
await conn.query<{ v: arrow.Int }>(`
    SELECT * FROM generate_series(1, 100) t(v)
`);

// ..., or fetch the result chunks lazely
for await (const batch of await conn.send<{ v: arrow.Int }>(`
    SELECT * FROM generate_series(1, 100) t(v)
`)) {
    // ...
}

// Close the connection to release memory
await conn.close();

  12. Configure the AWS S3 Filesystem

    main

    When interacting with AWS S3 via DuckDB-Wasm, you can provide an S3Config object to specify connection details. This is used within DuckDBFileInfo or DuckDBGlobalFileInfo to manage S3-specific file access.

    Available configuration keys:

    • region: The AWS region.
    • endpoint: The S3 endpoint URL.
    • accessKeyId: Your AWS access key ID.
    • secretAccessKey: Your AWS secret access key.
    • sessionToken: Your AWS session token.
    interface S3Config {
    region?: string;
    endpoint?: string;
    accessKeyId?: string;
    secretAccessKey?: string;
    sessionToken?: string;
}