aiochclient

repository·master·Indexed 42 days ago

https://github.com/maximdanilchenko/aiochclient

An asynchronous HTTP(s) client for ClickHouse designed for Python 3.10+. It features a fully typed interface, automatic bidirectional type conversion, and support for high-performance binary engines (Native and RowBinary). The client integrates with aiohttp or httpx and provides methods for executing DDL, fetching results via fetch(), fetchrow(), and fetchval(), and streaming large datasets using iterate().

Tokens
5.8K
Snippets
20
Records
28
Agent score
53%

What's inside aiochclient

  1. Access query results by name or index

    master

    Rows returned by fetch methods are lightweight, memory-efficient objects that support both name-based and index-based access. They also support mapping interfaces like .keys(), .values(), and slicing.

    row = await client.fetchrow("SELECT a, b FROM t WHERE a=1")
    
    assert row["a"] == 1          # Access by name
    assert row[0] == 1            # Access by index
    assert row[:] == (1, val)     # Slicing (returns tuple)
    assert list(row.keys()) == ["a", "b"]
    assert list(row.values()) == [1, val]
  2. Performance optimization via bulk column encoders

    master

    The aiochclient library uses type-specific bulk encoders within the encode_column method to optimize the serialization of data into RowBinary format. This reduces the overhead caused by per-value writers, especially for non-numeric scalar columns.

    Key optimized encoders include:

    • String: Uses write_string_column (implemented in Cython with a pure Python fallback) which writes varint-prefixed UTF-8 directly into a single bytearray. This is highly efficient for string-heavy datasets.
    • Date: Uses array("H", days) for encoding.
    • DateTime (no tz): Uses array("I", seconds) for encoding.
    • Array(String): Automatically benefits from the optimized String encoder as its flat child is encoded using the same logic.

    While these optimizations significantly speed up the encoding stage (up to 2.26x), the overall INSERT speed may remain limited by the ClickHouse server's processing capacity (server-bound). However, faster encoding reduces the cost of each streamed block, which is critical for high-throughput streaming and overlapping I/O operations.

  3. Use Native and RowBinary engines for high performance

    master

    By default, aiochclient uses the TSV (text) format. For significantly higher performance, you can opt-in to binary engines via ChClient constructor arguments:

    • native=True: Uses ClickHouse's column-oriented Native format. This is the fastest engine for bulk SELECT and INSERT operations. It decodes columns at once using the array module.
      • Note on Atomicity: Native INSERTs are streamed in blocks (default 8192 rows). If an error occurs during encoding, blocks already sent are committed. To ensure atomic inserts, set insert_block_size=0.
      • Note on Timezones: The Native format does not carry column timezones; DateTime values will return as naive UTC datetime objects.
    • binary=True: Uses the row-oriented RowBinary format. It is faster than TSV and is the best choice when using iterate() for row-by-row streaming.
    # Fastest for bulk operations
    client = ChClient(session, native=True)
    
    # Atomic Native inserts
    client_atomic = ChClient(session, native=True, insert_block_size=0)
    
    # Best for row-by-row streaming
    client_stream = ChClient(session, binary=True)
  4. Stream INSERT bodies using Native blocks for improved performance

    master

    To achieve higher throughput during INSERT operations, you can stream the request body as a sequence of Native blocks. This allows the client to encode the next block of rows while the server is simultaneously inserting the previous block, overlapping the encoding and network/server costs.

    Performance Gains

    Streaming the INSERT body in chunks (e.g., 4096 rows) can provide a performance boost of approximately 30–40% compared to sending a single large pre-encoded block.

    Important Trade-off: Atomicity

    An INSERT operation split into multiple blocks is not atomic in the event of a client-side encoding error. If an error occurs while encoding a later block, the earlier blocks may have already been successfully sent and processed by the server. This is generally not a concern for valid, uniformly-typed data, as an encoding error would likely have triggered a failure on the very first block.

  5. Optimize aiochclient throughput

    master

    To achieve maximum performance with aiochclient, follow these best practices:

    1. Use Binary Engines: Use native=True for bulk SELECT/INSERT and binary=True for iterate() streaming.
    2. Install Speedups: Use pip install aiochclient[*-speedups] to include ciso8601 for fast datetime parsing.
    3. Build Cython: Ensure the Cython extension is built during installation for a ~30% speed boost.
    4. Use uvloop: Use uvloop as the asyncio event loop policy.
  6. Connect to ClickHouse using ChClient

    master

    To use aiochclient, you must provide an existing aiohttp.ClientSession or httpx.AsyncClient instance to the ChClient constructor. You can verify the connection using the is_alive() method.

    from aiochclient import ChClient
    from aiohttp import ClientSession
    
    async def main():
        async with ClientSession() as s:
            client = ChClient(s)
            assert await client.is_alive()  # returns True if connection is Ok
  7. Quick Start: Check if ClickHouse is alive

    master

    To verify your connection, you can use ChClient with an aiohttp.ClientSession to call is_alive(). By default, this attempts to connect to localhost:8123 using the default user.

    import asyncio
    from aiochclient import ChClient
    from aiohttp import ClientSession
    
    async def main():
        async with ClientSession() as s:
            client = ChClient(s)
            alive = await client.is_alive()
            print(f"Is ClickHouse alive? -> {alive}")
    
    if __name__ == '__main__':
        asyncio.run(main())
  8. Reproduce aiochclient benchmarks

    master

    To reproduce the throughput comparisons against other Python ClickHouse clients, run the provided benchmark script. The benchmarks measure serialization/deserialization costs using a single connection without async parallelism to isolate CPU/encoding performance from IO concurrency.

    Benchmark Configuration:

    • Workload: A Memory engine table with mixed types (UInt32, Int64, Float64, String, Date, DateTime, Array(String), Nullable(Int32)).
    • SELECT: SELECT * of 50,000 rows, fully materialized into Python objects.
    • INSERT: Bulk-insert of 50,000 rows (table truncated before each run).
    • Metric: Throughput calculated as rows per best wall-clock time over 8 sequential runs.
    python benchmarks_vs_libs.py
  9. Install aiochclient with aiohttp or httpx

    master

    You can choose between aiohttp or httpx as the underlying HTTP connector. It is recommended to install the *-speedups extra for better performance, which includes ciso8601 for fast datetime parsing, and aiodns/faust-cchardet for aiohttp speedups.

    For aiohttp:

    pip install aiochclient[aiohttp]
    # Or with speedups:
    pip install aiochclient[aiohttp-speedups]

    For httpx:

    pip install aiochclient[httpx]
    # Or with speedups:
    pip install aiochclient[httpx-speedups]
    pip install aiochclient[aiohttp-speedups]
  10. Understand performance sensitivity to background load

    master

    Performance measurements for aiochclient (specifically throughput for columnar decoding) are sensitive to background CPU load. High background CPU usage (e.g., security scanners, virtualization overhead like Docker/QEMU) can significantly reduce absolute throughput numbers.

    When evaluating performance, note that:

    • The optimization for GC management provides a consistent performance boost regardless of system load.
    • Throughput figures reported in documentation (e.g., ~1.0M rows/sec) are typically achieved on idle machines. Under heavy load, these numbers will decrease, but the relative performance gain from the library's optimizations remains consistent.