ClickHouse JS Client

repository·main·Indexed 43 days ago

https://github.com/clickhouse/clickhouse-js

The official TypeScript client for ClickHouse DB, optimized for performance with zero external dependencies. It provides specialized implementations for both Node.js and Web environments, featuring support for streaming, async inserts, and RowBinary data decoding.

Tokens
85.4K
Snippets
229
Records
345
Agent score
56%

What's inside ClickHouse JS

  1. Use the ClickHouse Node.js client (@clickhouse/client)

    main

    The @clickhouse/client package is designed exclusively for Node.js runtimes (including Next.js Node runtime API routes, React Server Components, and Server Actions).

    Important: Do NOT use this package for browser components, Web Workers, Next.js Edge runtime, or Cloudflare Workers. For those environments, use @clickhouse/client-web instead.

    Core API Methods

    • client.insert(): Use this to write rows to ClickHouse.
    • client.query(): Use this to read data. You can process results using .json(), .text(), or .stream() on the returned resultSet.
    • client.command(): Use this for DDL and statements that do not return rows (e.g., CREATE, DROP, TRUNCATE, ALTER, or SET).
    • client.ping(): Performs a health check. It returns { success, error? } and never throws on connection failure.
    • client.exec(): Returns the raw response stream of an arbitrary statement (rarely needed for standard coding).
    import { createClient } from '@clickhouse/client';
    
    const client = createClient({ url: 'http://localhost:8123' });
    
    // Example usage
    await client.close();
  2. Understanding RowBinary skill evaluation results

    main

    This document provides an evaluation report for the RowBinary skill within the clickhouse-js ecosystem, specifically testing how different LLM models (Claude Haiku, Sonnet, and Opus) perform when using the skill versus when they are not provided with it.

    Key Metrics for Haiku 4.5

    • Pass rate (26 evals) with skill: 86.2%
    • Pass rate (26 evals) without skill: 53.2%
    • Skill Impact: The skill provides a +33pp (percentage point) lift for the Haiku model.

    Performance Characteristics

    • Baseline Gap: The skill is highly effective at closing the baseline gap for weaker models like Haiku, similar to its effect on Sonnet.
    • Ceiling Limitation: Unlike Opus or Sonnet, Haiku with the skill does not reach the same performance ceiling (~86% vs ~94%). It struggles with complex operational tasks, such as dynamic nested type-headers or applying specific optimizations to novel schemas.
    • Reliability: Haiku is highly reliable when the skill provides concrete, single-pattern guidance (e.g., varint readers or AggregateFunction handling), but it may fail to apply optimizations (like the no-bigint word-copy trick) if the schema does not match the provided example exactly.
  3. Explore ClickHouse JS client examples

    main

    The clickhouse-js repository contains a comprehensive suite of examples categorized by use case. These examples are available for both Node.js and Web environments, though some performance-oriented examples are Node-only due to dependencies on Node-specific APIs like node:fs or Node streams.

    Example Categories

    • coding/: Day-to-day client API usage including connection, configuration, pinging, basic CRUD operations, parameter binding, sessions, and data type handling.
    • performance/: High-throughput patterns such as streaming inserts/selects with backpressure, async inserts, and server-side bulk moves.
    • troubleshooting/: Methods to diagnose and recover from issues like timeouts, request cancellation, and server-side errors.
    • security/: Implementation of TLS (basic and mutual), Role-Based Access Control (RBAC), and SQL-injection-safe parameter binding.
    • schema-and-deployments/: DDL operations and connection strategies for different environments like Single-node, On-premise clusters, and ClickHouse Cloud.
  4. Available Node.js example categories

    main

    The Node.js examples are organized into the following functional categories:

    • coding/: Covers day-to-day API usage including connecting, configuring, pinging, basic INSERT/SELECT, parameter binding, sessions, data types, custom JSON handling, and OpenTelemetry tracing via tracer hooks.
    • performance/: Focuses on high-throughput operations like async inserts, streaming with backpressure, file/Parquet streams, progress streaming, and INSERT FROM SELECT.
    • troubleshooting/: Demonstrates handling abort/cancel, timeouts, long-running query progress, server error surfaces, and number-precision pitfalls.
    • security/: Covers TLS (basic and mutual), RBAC (roles and read-only users), and SQL-injection-safe parameter binding.
    • schema-and-deployments/: Provides CREATE TABLE examples for single-node, on-prem clusters, and ClickHouse Cloud, as well as column-shape features and deployment-specific connection strings.
  5. Scope of RowBinary decoding and encoding

    main

    Supported Decoding (Reading)

    Supports RowBinary, RowBinaryWithNames, and RowBinaryWithNamesAndTypes for Node.js. It supports both full-buffer and streaming (chunked) modes via:

    • advance() / NeedMoreData
    • readRows()
    • streamRowBatches() (includes a small-chunk warning and optional coalesceChunks() debounce filter).

    Supported Encoding (Writing)

    The inverse encode path is available via @clickhouse/rowbinary/writer. It provides writeX methods (mirroring readX) that append bytes to a Sink, as well as writeRows().

    Currently Out of Scope

    • Browsers and Edge runtimes.
    • Non-RowBinary formats (JSON, CSV, TSV, Parquet).
    • Big-endian hosts.
    • Certain decode-only paths (e.g., Dynamic, JSON, columnar typed-array path) are not yet mirrored in the writer.
  6. Understand the performance characteristics of RowBinary parsing

    main

    When deciding between JS and WASM for parsing RowBinary data, consider these performance realities:

    1. Parsing Speed vs. Memory Bandwidth

    JIT-compiled JS using DataView accessors already reads bytes at near-memory bandwidth (~16 GB/s for Float64). There is no significant headroom for a faster language to read the raw bytes faster.

    2. The Allocation Bottleneck

    In RowBinary decoding, approximately 83% of the time is spent on JS-side object and Date allocation, not on reading bytes. Because WASM cannot directly allocate JS objects (like Date, String, or BigInt) across the boundary, it cannot remove this primary bottleneck.

    3. The Columnar Advantage

    You can achieve a ~4x performance increase in JS by changing the output contract from row-based objects to columnar typed arrays. This avoids the overhead of per-row object materialization.

  7. How ClickHouse client spans are lifecycle-managed

    main

    The client invokes startActiveSpan for query, command, exec, insert, and ping operations.

    Span Lifecycle and Attributes

    1. Initialization: Spans are named based on the operation (e.g., clickhouse.query). Initial attributes include db.system.name, server.address, and server.port.
    2. Execution: The operation runs inside the provided callback. The client automatically attaches the query_id to the span once assigned.
    3. Completion:
      • On success, status is left UNSET.
      • On failure, the client records the exception, sets the error type, and sets the status to ERROR.
      • For command, exec, insert, and ping, the span ends when the method returns.

    The query() dual-span model

    Calling query() produces two distinct spans to separate the network round-trip from data consumption:

    • clickhouse.query: Covers the HTTP request. It starts when query() is called and ends as soon as response headers arrive.
    • clickhouse.query.stream: A child span covering the ResultSet lifetime. It starts after headers are received and ends when the result set is fully consumed, closed via close(), or fails. It carries metrics like clickhouse.response.decoded_bytes and db.response.returned_rows.
  8. How sessions and temporary tables work

    main

    In ClickHouse, temporary tables and session-scoped state (like SET commands) are tied to a session_id. Without a stable session_id provided across multiple calls, every request is treated as a fresh server-side session, causing temporary tables to disappear and SET settings to be lost.

    To maintain state across multiple calls, you must provide a session_id (e.g., using crypto.randomUUID()) either during client initialization via createClient or on a per-call basis.

    import { createClient } from "@clickhouse/client";
    import * as crypto from "node:crypto";
    
    const client = createClient({
      session_id: crypto.randomUUID(),
    });
    
    // This table will persist for subsequent calls using the same client/session_id
    await client.command({ query: "CREATE TEMPORARY TABLE my_temp_table (id Int32)" });
  9. Avoid using @clickhouse/client-common when building new clients

    main

    The package @clickhouse/client-common is deprecated.

    Historically, shared code between @clickhouse/client (Node.js) and @clickhouse/client-web lived in this package. To simplify versioning and avoid bundling issues, shared code is now inlined/co-located inside each runtime package (packages/client-node, packages/client-web, etc.).

    If you are building a new client:

    • Do not attempt to depend on @clickhouse/client-common.
    • Instead, copy the specific pieces of code you need directly into your own package, following the pattern used by client-node and client-web.
  10. Decide between RowBinary, JSON, and Native formats

    main

    Before implementing a RowBinary codec, ensure it is the most efficient format for your use case. Choosing the wrong format can lead to slower performance than standard JSON parsing.

    When to use JSON (e.g., JSONEachRow)

    Use JSON when your data consists mostly of strings or JSON-like values that you consume wholesale. V8's native JSON.parse is highly optimized and often faster than a custom JavaScript RowBinary decoder. To mitigate the wire cost of repetitive JSON keys, use HTTP response compression like gzip or zstd.

    When to use RowBinary

    RowBinary is superior when the data is dominated by:

    • Wide numerics: Int128, Int256, UInt128, UInt256, Decimal128, Decimal256.
    • Binary / fixed-width blobs: IPv4, IPv6, UUID, FixedString.
    • High-volume fixed-width numeric columns: Where values can be read directly via DataView.

    When to use Native

    Use the Native format if your primary goal is columnar loading or client-side analytics (e.g., feeding typed arrays to a Web Worker or WASM). Native is column-major, allowing data to be loaded directly into typed arrays without transposition.

  11. How RowBinary writing works in Node.js

    main

    The @clickhouse/rowbinary package allows you to encode JavaScript values into a RowBinary payload for ClickHouse.

    Core Concepts

    • Leaf Writers: Functions like writeUInt8 or writeString that encode a single value directly into a Sink.
    • Combinators: Functions like writeArray or writeTuple that take sub-writers and return a new writer, allowing you to compose complex types without per-element closures.
    • Sink: A caller-supplied byte buffer and current write offset. It maintains the state of the write operation but does not contain write methods itself.
    • Writer<T>: A type representing the encoding logic for a specific type T.

    To use these, you can import the barrel from @clickhouse/rowbinary/writer or import specific modules for individual types to reduce bundle size.