DOCSEARCHbyAlgolia
Submit Docs

Altinity Kubernetes Operator for ClickHouse

repository·master·Indexed 25 days ago

https://github.com/altinity/clickhouse-operator

A specialized Kubernetes operator designed to automate the lifecycle of ClickHouse clusters, including scaling, version upgrades, and configuration management. It utilizes Custom Resource Definitions (CRDs) to manage cluster state and supports customized storage, pod, and service templates. The operator includes integration for Prometheus metrics via ServiceMonitor and provides FIPS 140-3 compatible images with ACVP reproducibility harnesses for cryptographic validation.

Tokens
40.7K
Snippets
89
Records
167
Agent score
81%

What's inside altinity-clickhouse-operator

  1. Overview of Altinity Kubernetes Operator for ClickHouse

    master

    The Altinity Kubernetes Operator for ClickHouse automates the creation, configuration, and management of ClickHouse clusters on Kubernetes. It uses Custom Resources to define cluster state and provides features for scaling, version upgrades, and configuration management.

    Key Capabilities:

    • Cluster Management: Creates and manages ClickHouse clusters via custom resources.
    • Customization: Supports customized storage provisioning (via VolumeClaim templates), pod templates, and service templates for endpoints.
    • Configuration & Users: Manages ClickHouse configurations and user management.
    • Scaling & Upgrades: Handles cluster scaling (including automatic schema propagation) and ClickHouse version upgrades.
    • Observability: Exports ClickHouse metrics to Prometheus.
    • Security: FIPS 140-3 compatible operator and metrics-exporter images are available.

  2. Define ClickHouse cluster layouts and topologies

    master

    The ClickHouse cluster layout within a Custom Resource is defined in the .spec.clusters.layout section. This section determines how many shards and replicas are created and how they are distributed across pods. You can use simple dimensions (shardsCount, replicasCount) or advanced, explicit specifications using the shards and replicas arrays to customize specific pods, ports, or templates.

          - name: all-counts
        templates:
          podTemplate: clickhouse-v18.16.1
          dataVolumeClaimTemplate: default-volume-claim
          logVolumeClaimTemplate: default-volume-claim
        layout:
          shardsCount: 3
          replicasCount: 2

  4. ClickHouse Operator Kubernetes Resources

    master

    The ClickHouse operator installation creates the following Kubernetes resources:

    • ClickHouseInstallation (CRD): Extends the Kubernetes API with the ClickHouseInstallation kind, allowing you to manage ClickHouse clusters via this custom resource.
    • Service Account (clickhouse-operator): Provides an identity for the operator processes to authenticate with the Kubernetes API server.
    • Cluster Role Binding (clickhouse-operator): Grants the clickhouse-operator Service Account (in kube-system namespace) the cluster-admin role, providing cluster-wide permissions.
    • Deployment (clickhouse-operator): Runs the operator application within the kube-system namespace.

  5. Use ClickHouse macros for replication

    master

    The operator provides a set of macros that can be used within ClickHouse table definitions to automate replication paths and cluster identification.

    Operator-provided macros:

    • {installation}: The name of the ClickHouse Installation.
    • {cluster}: The primary cluster name.
    • {replica}: The replica name in the cluster (maps to the pod service name).
    • {shard}: The shard ID.

    Standard ClickHouse macros:

    • {database}: The current database.
    • {table}: The current table.

  6. How the ACVP wrapper dispatch works

    master

    The ACVP responder is triggered using an argv[0] dispatch pattern. A single binary built with the acvp_wrapper build tag can act as either the standard operator or the ACVP responder depending on its filename:

    1. Standard Operator: Running the binary as clickhouse-operator executes the normal operator logic with no ACVP code paths active.
    2. ACVP Responder: Running the binary with a basename ending in -acvp (e.g., clickhouse-operator-acvp) triggers the ACVP responder logic.

    This pattern is used for both the clickhouse-operator and the metrics-exporter (which produces metrics-exporter-acvp). Each binary statically links its own copy of the Go FIPS module.

  7. Understand the security hardening axes

    master

    The operator's security posture is divided into three orthogonal (independent) axes. You can enable any combination of these:

    1. Transport Hardening (security.policy): Controls TLS verification, IPC mode, and scheme coercion (e.g., forcing HTTPS) for outbound clients (ClickHouse, ZooKeeper, Kubernetes).
    2. Cryptographic-module Gate (security.fips.enforced): Asserts that the operator binary is linked with the Go FIPS 140-3 module and running with appropriate environment variables (GODEBUG=fips140=on).
    3. Workload Supply-chain Gate (security.images.policy): Ensures that every ClickHouse and Keeper container image is FIPS-compliant by checking image tags and SELECT version() output.

    Posture Matrix

    security.policysecurity.fips.enforcedOperator Posture
    Permissive (default)false (default)Pre-0.27.1 behavior. No coercion, no FIPS gates.
    EnforcedfalseTLS-only hardening.
    PermissivetrueFIPS module gate only.
    EnforcedtrueFull operator-side FIPS posture.

  8. Automatic TLS detection for ClickHouseKeeper

    master

    The operator automatically detects if the referenced keeper requires TLS by inspecting the service port specification. You do not need to manually configure secure connections if the ports match these patterns:

    • Insecure: Port 2181 or a port named zk.
    • Secure: Port 2281 or a port named zk-secure. If detected, the operator automatically sets <secure>1</secure> in the ClickHouse configuration and sets secure: true for the resolved ZooKeeper nodes.

  9. Configure Pod Templates and Zoned Layouts

    master

    The .spec.templates.podTemplates section allows you to define templates for ClickHouse Pods. This is useful for managing multiple versions or specific hardware requirements. Multiple pod templates enable smooth version updates.

    Zoned Layouts

    The zone and distribution fields work together to define a zoned layout of ClickHouse instances across nodes. This is a shortcut that automatically configures Kubernetes affinity.nodeAffinity and affinity.podAntiAffinity.

    • zone: Defines the target nodes. You can specify a key and values (to match node labels) or just values (for cloud-specific availability zones).
    • distribution: Controls how pods are spread. Common value: OnePerHost (ensures one ClickHouse instance per physical/virtual host).
      templates:
    podTemplates:
      - name: clickhouse-v18.16.1
        zone:
          key: "clickhouse"
          values:
            - "allow"
        distribution: "OnePerHost"
        spec:
          containers:
            - name: clickhouse
              image: clickhouse/clickhouse-server:24.8
              volumeMounts:
                - name: default-volume-claim
                  mountPath: /var/lib/clickhouse
              resources:
                requests:
                  memory: "64Mi"
                  cpu: "100m"
                limits:
                  memory: "64Mi"
                  cpu: "100m"

  10. Enable Transport Hardening with `security.policy: Enforced`

    master

    Setting security.policy: Enforced acts as a master switch that tightens the operator's outbound TLS posture by overriding user settings with strict values.

    When Enforced is active, the following happens:

    1. Automatic Coercion: The following fields are forced to their strict positions (logged at INFO level):

      • security.clickhouse.tls.verify $\rightarrow$ Strict
      • security.clickhouse.tls.minVersion $\rightarrow$ 1.3
      • security.zookeeper.tls.verify $\rightarrow$ Strict
      • security.zookeeper.tls.minVersion $\rightarrow$ 1.3
      • security.kubernetes.tls.verify $\rightarrow$ Strict
      • security.kubernetes.tls.minVersion $\rightarrow$ 1.3
      • security.ipc.mode $\rightarrow$ Secure

    2. Scheme Coercion: clickhouse.access.scheme: http is automatically coerced to https to prevent unencrypted dialing.

    3. ZooKeeper Enforcement: The operator rejects CHIs that reference plain-text external ZooKeeper. Every node in spec.configuration.zookeeper.nodes[] must have secure: true. If missing, the CHI will enter status: Aborted with the reason [FIPSValidationFailed].

    4. Auth Restriction: The operator rejects ZooKeeper digest: authentication files (which use SHA-1).

  11. ACVP Wrapper Compliance Scope and Context

    master

    The ACVP wrapper in this repository is a reproducibility harness for ACVP expected-output tests. It is not a CMVP certificate artifact itself.

    Key Concepts

    • Compliance Source: The actual cryptographic implementation being validated is Go's native crypto/internal/fips140/... module when executed with the environment variable GODEBUG=fips140=on (or only).
    • Validation Tracking: Go's FIPS validation lifecycle is managed via CMVP (Cryptographic Module Validation Program) lists.
    • Configuration: The repository uses pkg/util/fips/acvp/acvp_test_fips140v1.26.public.config.json for public-Go-API validation. This configuration intentionally excludes ML-KEM and ML-DSA vector suites because those paths require internal Go crypto APIs not exposed in Go 1.26.

  12. How schema auto-creation works during scaling

    master

    When you scale up a ClickHouse cluster, the clickhouse-operator automatically manages schema synchronization to ensure new instances are consistent with the existing cluster. The behavior depends on whether you are adding a shard or a replica:

    Adding a Shard

    When a new shard is added, the operator:

    1. Analyzes existing shards in the cluster for distributed and corresponding local tables.
    2. Creates the necessary databases for both local and distributed tables.
    3. Creates the local tables.
    4. Creates the distributed tables.

    Adding a Replica

    When a new replica is added, the operator:

    1. Analyzes other replicas at the same shard for replicated tables.
    2. Creates the necessary databases for replicated tables.
    3. Creates the replicated tables.
    4. Proceeds with the shard-addition logic (creating local and distributed tables) to ensure full cluster consistency.