DOCSEARCHbyAlgolia
Submit Docs

ClickHouse Documentation Tools

repository·main·Indexed 42 days ago

https://github.com/clickhouse/clickhouse-docs

Internal tooling and scripts for managing ClickHouse documentation, including Algolia search indexing via index_pages.py, search performance evaluation with compute_ndcg.py, glossary term wrapping, markdownlint custom rules, and automated translation workflows for multiple locales.

Tokens
295K
Snippets
674
Records
1.2K
Agent score
51%

What's inside ClickHouse

  1. Overview of ClickHouse Rust client capabilities

    main

    The ClickHouse Rust client is designed for high-performance integration using the following principles:

    • Serialization: Uses serde for encoding and decoding rows. It supports standard serde attributes like skip_serializing, skip_deserializing, and rename.
    • Transport: Uses the RowBinary format over HTTP. (Note: Plans exist to support Native over TCP).
    • Compression: Supports LZ4 compression/decompression.
    • Security: Supports TLS via native-tls or rustls-tls.
    • Functionality: Provides APIs for selecting/inserting data, executing DDLs, client-side batching (via inserter), and WATCH functionality.

  2. What is Langfuse?

    main
    Langfuse is an open-source LLM engineering platform designed for debugging, analyzing, and iterating on LLM applications. It is part of the ClickHouse ecosystem and uses ClickHouse as its core observability backend to provide high-performance, scalable storage for traces and events. This architecture allows Langfuse to handle billions of traces with low latency using ClickHouse's columnar storage and analytical capabilities.

  3. Overview of chDB

    main

    chDB is an in-process SQL OLAP Engine powered by ClickHouse (v25.8.2.1). It allows you to use ClickHouse's analytical capabilities directly within your programming language without the need to install or connect to a separate ClickHouse server.

    Key features include:

    • In-process execution: No external database installation required.
    • Wide format support: Native input and output for Parquet, CSV, JSON, Arrow, ORC, and 70+ other formats.
    • High performance: Minimized data copying between C++ and Python using python memoryview.
    • Ecosystem integration: Native support for Pandas, Arrow, and DB API 2.0.
    • DataStore API: A Pandas-compatible API that uses SQL optimization and lazy evaluation to speed up operations.

  4. Overview of the ClickHouse HTTP Interface

    main

    The HTTP interface provides REST API access to ClickHouse, allowing interaction from any platform or programming language. While more limited than the native interface, it offers broader language support.

    Default Ports:

    • 8123: HTTP
    • 8443: HTTPS (can be enabled)

    Basic Connectivity:

    • A GET / request returns Ok. (status 200).
    • For health checks, use GET /ping, which returns Ok. (available since version 18.12.13).
    • To check replica delay, use /replicas_status.
    $ curl 'http://localhost:8123/'
Ok.

$ curl 'http://localhost:8123/ping'
Ok.

$ curl 'http://localhost:8123/replicas_status'
Ok.

  5. Overview of ClickHouse Deployment Modes

    main

    ClickHouse can be deployed in five distinct ways, all utilizing the same underlying database engine. The choice of deployment depends on whether you need a production-grade server, a managed cloud service, a command-line utility, a standalone local processor, or an embedded engine for applications.

    Deployment Options:

    • ClickHouse Server: Traditional client/server architecture for production, on-premises, or cloud-based (AWS, GCP, Azure) deployments. Supports distributed clusters for high availability.
    • ClickHouse Cloud: A fully managed service that includes integrated tooling like ClickPipes (data ingestion), a dedicated querying API, SQL Console dashboarding, and built-in query insights for monitoring.
    • ClickHouse CLI (clickhousectl): A tool for local development (managing local server instances) and ClickHouse Cloud management (scaling services, managing API keys).
    • clickhouse-local: A standalone executable for command-line data processing and ad-hoc analysis of local or cloud-stored files (CSV, JSON, Parquet, etc.) without running a server.
    • chDB: An in-process embedded database engine (available for Python, Go, Rust, NodeJS, and Bun) that allows you to run ClickHouse queries directly within your application process.

  6. Langfuse Evaluation & Dataset Features

    main

    Langfuse provides tools to ensure the quality and reliability of LLM applications through various evaluation methods:

    • Analytics: Plot evaluation results in the Langfuse Dashboard.
    • User Feedback: Collect feedback via the Browser SDK (frontend) or SDKs/API (server-side).
    • LLM-as-a-Judge: Run managed LLM-based evaluations on production or development traces, including step-wise evaluations.
    • Experiments: Evaluate prompts and models against datasets directly in the UI without custom code.
    • Annotation Queue: Use human annotations to baseline your evaluation workflow.
    • Custom Evaluations: Add custom numeric, boolean, or categorical scores via the API or SDKs.

  7. Monitor ClickHouse Cloud deployments

    main

    ClickHouse Cloud provides several methods for monitoring service health, resource utilization, and query performance. Choose a method based on your requirements for setup effort and system impact:

    • Cloud Console dashboards: Built-in dashboards for day-to-day monitoring. Requires no setup and does not wake idle services.
    • Notifications: Alerts for scaling events, errors, mutations, and billing. Requires customization but does not wake idle services.
    • Prometheus endpoint: Export metrics to Grafana, Datadog, or other Prometheus-compatible tools. Requires an API key and scraper configuration; does not wake idle services.
    • System table queries: Deep debugging and custom analysis via direct SQL queries against system tables. Warning: This method adds query load to the system and prevents ClickHouse Cloud instances from idling, which can increase costs. It also relies on the production system being operational.
    • Community and partner integrations: Includes Datadog agent integration and the Billing & Usage API. Setup varies by tool.
    • Advanced dashboard reference: Detailed reference for advanced dashboard visualizations and troubleshooting.

  8. Langfuse Prompt Management Features

    main

    Langfuse allows you to manage, version, and optimize prompts throughout the development lifecycle using the following capabilities:

    • Create & Version Control: Create prompts via UI, SDKs, or API, and collaboratively version/edit them.
    • Deployment: Deploy prompts to production or specific environments using labels, enabling updates without code changes.
    • Metrics: Compare latency, cost, and evaluation metrics across different prompt versions.
    • Playground: Test prompts instantly in the Langfuse playground.
    • Trace Linking: Link prompts with traces to observe real-world performance in context.
    • Change Tracking: Monitor how prompts evolve over time.

  9. Langfuse Observability Features

    main

    Langfuse provides tracing capabilities to monitor and debug non-deterministic LLM applications. Key observability features include:

    • Trace Details: Track every LLM call and relevant logic within your application.
    • Sessions: Track multi-step conversations or agentic workflows.
    • Timeline: Inspect the timeline view to debug latency issues.
    • Users: Monitor costs and usage per user by providing a userId.
    • Agent Graphs: Visualize LLM agents as graphs to illustrate complex agentic workflows.
    • Dashboard: Monitor quality, cost, and latency metrics via a centralized dashboard.

  10. Compare ClickHouse Cloud service tiers

    main

    ClickHouse Cloud offers three main service tiers tailored to different workload requirements:

    Basic Tier

    Best for: Departmental use cases with smaller data volumes and no hard reliability guarantees.

    • Configuration Example: 1 replica, 8 GiB RAM, 2 vCPU.
    • Included in base example: 500 GB compressed data, 500 GB backup, 10 GB public internet egress, 5 GB cross-region transfer.

    Scale Tier

    Best for: Workloads requiring enhanced SLAs (2+ replicas), scalability, and advanced security.

    • Features: Always-on, auto-scaling (with configurable maximums to prevent runaway costs).
    • Configuration Example: 2 replicas, 8 GiB RAM, 2 vCPU.
    • Included in base example: 100 GB public internet egress, 10 GB cross-region transfer.

    Enterprise Tier

    Best for: Large-scale, mission-critical deployments with stringent security and compliance needs.

    • Features: Always-on, high-performance compute configurations.
    • Included in base example: 1 TB public internet egress, 500 GB cross-region transfer.

  11. Cloud access management overview

    main

    ClickHouse Cloud provides several mechanisms for managing security and access. Depending on your role, you can manage personal account security, administrator-level user access, custom roles, or automated identity provisioning via SAML and SCIM.

    Key Management Areas:

    • Personal Account Security: Manage your own credentials, passwords, Multi-Factor Authentication (MFA), and account recovery.
    • User & Role Management: Administrators can manage cloud users via the console, create custom roles, and manage database-level users or SQL console role assignments.
    • Identity & Access Management (IAM) Integrations:
      • SAML SSO: Configure and troubleshoot SAML-based Single Sign-On.
      • SCIM Provisioning: Automate user and group lifecycles using SCIM with providers like Okta or Microsoft Entra ID.
    • RBAC Migration: Guidance for migrating to the new role-based access control (RBAC) system using custom roles.
    • Permission Verification: Use SQL queries to set up and verify database permissions.

  12. Integrate ClickHouse with Power BI

    main

    Microsoft Power BI can query or load data from ClickHouse Cloud or self-managed deployments. The integration typically involves using the ClickHouse Connector in Power BI Desktop for visualization and an on-premise data gateway for the Power BI Service.

    There are two primary workflows:

    1. Using the ClickHouse Connector (Recommended): Uses the built-in connector in Power BI Desktop (available from version 2.137.751.0). This is preferred for DirectQuery mode.
    2. Using the ODBC Driver: Useful for specific import-only use cases or when connecting via an on-premise data gateway using a User DSN.