clickhouse-cpp

repository·master·Indexed 18 days ago

https://github.com/clickhouse/clickhouse-cpp

A C++17 client library for ClickHouse using the native protocol. It provides a direct API for executing SQL queries and performing high-performance columnar data insertions and selections. Features include support for batch insertions via BeginInsert/SendInsertBlock, asynchronous inserts, and SelectWithExternalData for temporary tables. The library can be integrated via CMake (git submodule or FetchContent) or Bazel.

Tokens
8.4K
Snippets
25
Records
35
Agent score
62%

What's inside clickhouse-cpp

  1. Customize GoogleTest via the custom directory

    master
    The contrib/gtest/include/gtest/internal/custom/ directory serves as an injection point for user-defined configurations. By defining specific macros, you can override core GoogleTest behaviors such as stack trace generation, temporary directory management, logging, threading primitives, and symbol exporting. This is useful when porting GoogleTest to specialized environments or integrating it with existing system libraries.
  2. Override core GoogleTest behaviors in `gtest.h`

    master

    You can customize fundamental GoogleTest operations by defining the following macros in the custom directory:

    • GTEST_OS_STACK_TRACE_GETTER_: Provide the name of an implementation of OsStackTraceGetterInterface to customize how stack traces are retrieved.
    • GTEST_CUSTOM_TEMPDIR_FUNCTION_: Provide an override for testing::TempDir(). The override must match the signature and semantics of testing::TempDir.
  3. Implement retries with clickhouse::Client

    master

    When implementing retry logic for clickhouse::Client:

    1. If a previous attempt threw an exception, call clickhouse::Client::ResetConnection() before retrying.
    2. For clickhouse::Client::Insert(), you can reuse the existing Block from the previous attempt instead of rebuilding it.
  4. Include clickhouse-cpp using CMake git submodule

    master

    Add clickhouse-cpp as a git submodule (e.g., in contrib/clickhouse-cpp) and include it in your CMakeLists.txt. Ensure you set the compatibility flags CH_USE_ABSEIL_FOR_BIGNUM and CH_MAP_BOOL_TO_UINT8 to OFF.

    cmake_minimum_required(VERSION 3.13)
    project(application-example LANGUAGES CXX)
    
    set(CH_USE_ABSEIL_FOR_BIGNUM OFF)
    set(CH_MAP_BOOL_TO_UINT8 OFF)
    
    add_subdirectory(contrib/clickhouse-cpp)
    
    add_executable(application-example app.cpp)
    target_link_libraries(application-example PRIVATE clickhouse-cpp-lib)
  5. Include clickhouse-cpp using CMake FetchContent

    master

    Use CMake's FetchContent to automatically download and configure the library during the configuration step. Set the compatibility flags to OFF as well.

    cmake_minimum_required(VERSION 3.14)
    project(application-example LANGUAGES CXX)
    
    include(FetchContent)
    
    set(CH_USE_ABSEIL_FOR_BIGNUM OFF)
    set(CH_MAP_BOOL_TO_UINT8 OFF)
    
    FetchContent_Declare(
        clickhouse_cpp
        GIT_REPOSITORY https://github.com/ClickHouse/clickhouse-cpp.git
        GIT_TAG v2.6.2
    )
    FetchContent_MakeAvailable(clickhouse_cpp)
    
    add_executable(application-example app.cpp)
    target_link_libraries(application-example PRIVATE clickhouse-cpp-lib)
  6. Use asynchronous inserts via SQL settings

    master

    Asynchronous inserts in ClickHouse work best when using SQL text format. You can enable them by specifying settings in the INSERT query or by using query.SetSetting().

    Strong Recommendation: Use async_insert=1 and wait_for_async_insert=1 to ensure the client is aware of errors and to prevent server overload.

    // Option 1: Using SETTINGS clause in SQL
    clickhouse::Query query("INSERT INTO default.test SETTINGS async_insert=1,wait_for_async_insert=1,async_insert_busy_timeout_ms=5000,async_insert_use_adaptive_busy_timeout=0,async_insert_max_data_size=104857600 VALUES(10,10)");
    client.Execute(query);
    
    // Option 2: Using SetSetting
    clickhouse::Query query("INSERT INTO default.test VALUES(10,10)");
    query.SetSetting("async_insert", clickhouse::QuerySettingsField{ "1", 1 });
    query.SetSetting("wait_for_async_insert", clickhouse::QuerySettingsField{ "1", 1 }); // strong recommendation
    query.SetSetting("async_insert_busy_timeout_ms", clickhouse::QuerySettingsField{ "5000", 1 });
    query.SetSetting("async_insert_max_data_size", clickhouse::QuerySettingsField{ "104857600", 1 });
    query.SetSetting("async_insert_use_adaptive_busy_timeout", clickhouse::QuerySettingsField{ "0", 1 });
    client.Execute(query);
  7. Build clickhouse-cpp from source

    master

    To build the library using CMake, use the following recommended settings. It is highly recommended to disable the legacy defaults CH_USE_ABSEIL_FOR_BIGNUM and CH_MAP_BOOL_TO_UINT8 for new projects to ensure future compatibility.

    $ mkdir build .
    $ cd build
    $ cmake .. -DCH_USE_ABSEIL_FOR_BIGNUM=NO -DCH_MAP_BOOL_TO_UINT8=NO
    $ make
  8. Perform batch insertions with BeginInsert/SendInsertBlock

    master

    To manage large datasets without high memory usage, use the batch insertion pattern. This involves starting an insertion with an INSERT statement ending in VALUES, appending data to a Block, sending it via SendInsertBlock, and finally calling EndInsert.

    // Start the insertion.
    auto block = client->BeginInsert("INSERT INTO foo (id, name) VALUES");
    
    // Grab the columns from the block.
    auto col1 = block[0]->As<ColumnUInt64>();
    auto col2 = block[1]->As<ColumnString>();
    
    // Add a couple of records to the block.
    col1.Append(1);
    col1.Append(2);
    col2.Append("holden");
    col2.Append("naomi");
    
    // Send those records.
    block.RefreshRowCount();
    client->SendInsertBlock(block);
    block.Clear();
    
    // Add another record to the block.
    col1.Append(3);
    
    // Send it and finish.
    block.RefreshRowCount();
    client->EndInsert();
  9. Include clickhouse-cpp using Bazel

    master

    Add clickhouse-cpp to your MODULE.bazel file and depend on @clickhouse-cpp//:clickhouse in your BUILD.bazel.

    Note: Bazel support is experimental. By default, it uses BoringSSL for TLS. You can switch to OpenSSL using --@clickhouse-cpp//:tls=openssl or disable TLS with --@clickhouse-cpp//:tls=no.

    # MODULE.bazel
    bazel_dep(name = "clickhouse-cpp", version = "2.6.2")
    # BUILD.bazel
    cc_binary(
        name = "application-example",
        srcs = ["app.cpp"],
        deps = ["@clickhouse-cpp//:clickhouse"],
    )
  10. Include clickhouse-cpp in your project using CMake FetchContent

    master

    The recommended way to integrate the library is using CMake's FetchContent module. This allows you to pin a specific version and build it as part of your project's workflow.

    To enable TLS/SSL support (required for ClickHouse Cloud), you must set the WITH_OPENSSL CMake option to YES. This requires OpenSSL development packages to be installed on your system:

    • Debian/Ubuntu: libssl-dev
    • Fedora/Red Hat: openssl-devel
    • macOS: openssl (via homebrew)

    After making the dependency available, link your target against clickhouse-cpp-lib.

    include(FetchContent)
    
    set(WITH_OPENSSL YES CACHE BOOL "Enable OpenSSL in clickhouse-cpp" FORCE)
    FetchContent_Declare(
        clickhouse-cpp
        GIT_REPOSITORY https://github.com/ClickHouse/clickhouse-cpp.git
        GIT_TAG v2.6.0   # can also be `master` or other banch
    )
    FetchContent_MakeAvailable(clickhouse-cpp)
    
    target_link_libraries(your-target PRIVATE clickhouse-cpp-lib)
  11. Execute a simple SELECT query

    master

    Use ch::Client to connect to a host and iterate through results using BeginSelect() and NextBlock(). Data is retrieved as columnar Block objects.

    #include <clickhouse/client.h>
    #include <iostream>
    
    namespace ch = clickhouse;
    
    int main()
    {
        ch::Client client{ch::ClientOptions{}.SetHost("localhost")};
    
        client.BeginSelect("SELECT 'Hello from ClickHouse :) ");
        
        while (auto block = client.NextBlock()) {
            auto col_msg = block->At(0)->AsStrict<ch::ColumnString>();
            
            for (size_t i = 0; i < block->GetRowCount(); ++i) {
                std::cout << col_msg->At(i) << "\n";
            }
        }
    }