DOCSEARCHbyAlgolia
Submit Docs

clickhouse-go

repository·main·Indexed 25 days ago

https://github.com/clickhouse/clickhouse-go

A high-performance Golang client for ClickHouse supporting both a native driver for optimal performance and a standard database/sql interface for compatibility. It supports TCP and HTTP transport, DSN-based configuration, TLS/SSL connections, and various compression algorithms including lz4, zstd, and gzip. Key features include async inserts via WithAsync(), batch management with PrepareBatch, and experimental support for arbitrary input/output formats over HTTP.

Tokens
17.5K
Snippets
16
Records
119
Agent score
86%

What's inside clickhouse-go

  1. Understand the relationship with ch-go

    main

    Versions of clickhouse-go >= 2.3.x utilize ch-go for low-level encoding/decoding.

    • ch-go: Provides a high-performance columnar interface suitable for performance-critical use cases.
    • clickhouse-go: Provides more familiar row-oriented and database/sql semantics at a slight performance cost compared to pure columnar access.

    Both approaches are supported by ClickHouse.

  2. Understand Golang type support for ClickHouse columns

    main

    The clickhouse-go driver supports implicit conversions between ClickHouse column types and Golang types to reduce the need for manual type alignment. Support is categorized into two phases:

    1. Insertion (Append/AppendRow): How Golang types are converted when writing data to ClickHouse.
    2. Read time (Scan): How ClickHouse data types are converted into Golang types when retrieving results.

    If a specific conversion is required but not supported, users are encouraged to raise an issue in the repository.

  4. Choose between clickhouse (native) and database/sql interfaces

    main

    The library provides two primary interfaces. Use the native interface for performance-sensitive work and the standard database/sql interface for compatibility with existing Go tooling or ORMs.

    Featureclickhouse.Open (native)sql.Open / clickhouse.OpenDB (std)
    PerformanceFaster (direct column encoding)Slower
    APIdriver.Conn (ClickHouse-specific)Standard database/sql
    Use whenNew code, performance-sensitiveExisting database/sql tooling, ORMs

    Both interfaces support TCP and HTTP transport.

  5. Configure TLS/SSL connections

    main

    All client connection methods (DSN, OpenDB, Open) use the Go tls package. To use TLS, the Options struct must contain a non-nil TLS pointer (*tls.Config).

    Setting secure in a DSN creates a minimal tls.Config with InsecureSkipVerify: false. This is typically sufficient for the secure native port (default 9440).

    If you need to connect to an IP address but your certificate only contains a DNS name in the Subject Alternative Name (SAN), set tls_server_name in the DSN or tls.Config.ServerName in your code.

  6. Configure HTTP protocol and Proxy

    main

    To use the HTTP protocol, modify the DSN to use http:// or set the Protocol field to clickhouse.HTTP in clickhouse.Options.

    Using DSN with Proxy: http://host1:8123,host2:8123/database?dial_timeout=200ms&max_execution_time=60&http_proxy=http%3A%2F%2Fproxy%3A8080

    Using clickhouse.OpenDB with Proxy: Set the HTTPProxyURL field in clickhouse.Options or use the HTTP_PROXY / HTTPS_PROXY environment variables.

    conn := clickhouse.OpenDB(&clickhouse.Options{
	Addr: []string{"127.0.0.1:8123"},
	Auth: clickhouse.Auth{
		Database: "default",
		Username: "default",
		Password: "",
	},
	Settings: clickhouse.Settings{
		"max_execution_time": 60,
	},
	DialTimeout: 30 * time.Second,
	Compression: &clickhouse.Compression{
		Method: clickhouse.CompressionLZ4,
	},
	Protocol:  clickhouse.HTTP,
})

  7. Run ClickHouse-go benchmarks

    main

    To measure performance on your own hardware, you can run the provided benchmark programs directly using go run. Alternatively, you can run the Go benchmark tests using the standard go test command.

    Example for running a specific benchmark program: go run benchmark/v2/read/main.go

    go run benchmark/v2/read/main.go

  8. Configure Compression

    main

    Compression is supported over both native and HTTP protocols.

    Native Protocol: Supports lz4, lz4hc, and zstd.

    HTTP Protocol:

    • HTTP web compression (whole request/response body): Uses gzip, deflate, or br. Controlled by the enable_http_compression setting. In clickhouse-go, this is used when Compression.Method is set to one of these three.
    • ClickHouse native block compression over HTTP: Uses lz4 or zstd. This uses ClickHouse HTTP query parameters compress=1 and decompress=1. In clickhouse-go, this is used when Compression.Method is lz4 or zstd.

    Warning: Avoid enabling both layers simultaneously to prevent unnecessary CPU usage from double compression.

    When using a DSN, you can enable compression via the compress parameter by specifying an algorithm name (zstd, lz4, lz4hc, gzip, deflate, br) or using true as a shorthand for lz4.

  9. Handle mid-stream exceptions in HTTP streams

    main

    When streaming results via HTTP, ClickHouse may encounter errors mid-stream. The clickhouse-go driver handles this by scanning the decompressed response body for an __exception__ marker.

    To ensure reliable error detection:

    1. The driver uses an exceptionScanReader to look for the __exception__ frame.
    2. On modern servers, this scan is validated against the X-ClickHouse-Exception-Tag header to prevent false positives from actual data.
    3. If an exception is found, the driver parses the error and returns it as a terminal error from the Read method.

    If you use the wait_end_of_query setting, the server buffers the entire result before responding. In this mode, failures arrive as non-200 HTTP status codes, and the in-band exception scan is skipped.

  10. Run ClickHouse server via Docker Compose

    main

    You can use the provided docker-compose.yml to spin up a ClickHouse server instance for local development or testing. The configuration exposes the HTTP interface on port 8123 and the native protocol interfaces on ports 9000 and 9009 (mapped to 127.0.0.1).

    Key environment variables and settings:

    • CLICKHOUSE_VERSION: Can be used to specify the server version (defaults to 25.12-alpine if not provided).
    • CLICKHOUSE_SKIP_USER_SETUP: Set to 1 to skip default user setup processes.
    services:
  clickhouse:
    environment:
      CLICKHOUSE_SKIP_USER_SETUP: 1
    networks:
      - clickhouse
    hostname: clickhouse
    container_name: clickhouse
    image: 'clickhouse/clickhouse-server:${CLICKHOUSE_VERSION-25.12-alpine}'
    ports:
      - 127.0.0.1:8123:8123
      - 127.0.0.1:9000:9000
      - 127.0.0.1:9009:9009
    healthcheck:
      test: ["CMD", "wget", "--no-verbose", "--tries=1", "--spider", "http://127.0.0.1:8123/ping"]
      interval: 10s
      timeout: 5s
      retries: 5
networks:
  clickhouse: null

  11. Explore native interface examples

    main

    The clickhouse interface (formerly native) provides high-performance access to ClickHouse. Available examples include:

    • Batching: batch.go, batch_release_connection.go, append_struct.go (batch struct), columnar_insert.go (columnar)
    • Async Inserts: async_native.go (native), async_http.go (http)
    • Querying: query_parameters.go, bind.go (deprecated), scan_struct.go (scan struct)
    • Advanced Features: multi_host.go (failover), ephemeral_native.go (native), ephemeral_http.go (http), client_info.go, geo.go, json_structs.go, variant.go, dynamic.go, bfloat16.go, qbit.go.

  12. Explore database/sql interface examples

    main

    The database/sql interface provides standard Go SQL semantics. Available examples include:

    • Connection: connect.go (open db)
    • Batching: batch.go
    • Async Inserts: async_native.go (native), async_http.go (http)
    • Querying: query_parameters.go, bind.go (deprecated)
    • Advanced Features: multi_host.go (failover), ephemeral_native.go (native), ephemeral_http.go (http), client_info.go, geo.go, dynamic.go, variant.go, bfloat16.go, qbit.go.