DOCSEARCHbyAlgolia
Submit Docs

Strimzi Kafka Operator Documentation

repository·main·Indexed 27 days ago

https://github.com/strimzi/strimzi-kafka-operator

An operator for running Apache Kafka clusters on Kubernetes and OpenShift. Documentation covers deployment via Helm, management of Kafka clusters, KRaft support, Kafka Connect, MirrorMaker, and the HTTP Kafka Bridge. Includes guides on upgrading the operator and CRDs, configuring securityContext, and using the Strimzi Drain Cleaner utility to maintain cluster health during node draining.

Tokens
276.3K
Snippets
495
Records
1.3K
Agent score
90%

What's inside Strimzi

  1. Overview of distributed tracing in Strimzi

    main

    Strimzi provides built-in support for distributed tracing to enable end-to-end tracking of messages from source systems, through Kafka, to target systems and applications. This allows you to monitor performance and investigate issues across microservices.

    Supported Kafka components for tracing include:

    • MirrorMaker: Traces messages from a source cluster to a target cluster.
    • Kafka Connect: Traces messages consumed and produced by Kafka Connect.
    • HTTP Bridge: Traces messages between Kafka and HTTP client applications.

    Note: Tracing is not supported for Kafka brokers.

  2. Overview of the Strimzi Access Operator

    main

    The Strimzi Access Operator is an optional feature designed to simplify sharing Kafka connection information and credentials across different Kubernetes namespaces.

    Instead of manually replicating secrets, the Access Operator manages the distribution of connection details via a Secret resource. This secret contains cluster connection details and optional user credentials, which can then be referenced by applications residing in any namespace.

  3. Overview of Strimzi Supported Features

    main

    The Strimzi Helm chart provides a comprehensive suite of features for managing Apache Kafka on Kubernetes:

    • Kafka Cluster Management: Deploys and manages all components, including dependencies like Apache ZooKeeper.
    • KRaft Support: Enables running Kafka in KRaft mode (without ZooKeeper).
    • Kafka Connect: Configures data sources and sinks.
    • Topic Management: Creates and manages Kafka Topics.
    • User Management: Creates and manages Kafka Users.
    • Connector Management: Creates and manages Kafka Connect connectors.
    • Kafka MirrorMaker: Mirrors data between different Kafka clusters.
    • HTTP Kafka Bridge: Allows HTTP-based message interaction.
    • Cruise Control: Automates partition balancing.
    • Auto-rebalancing: Automatically rebalances the cluster during scale-up or scale-down operations.
    • Tiered Storage: Offloads older data to lower-cost storage (e.g., object storage).
    • Monitoring & Observability: Built-in support for Prometheus monitoring and Grafana dashboards (via grafana_sidecar).

  4. Overview of Kafka System Tests

    main

    The Kafka system tests validate core Apache Kafka functionality within the Strimzi ecosystem. They are designed to ensure the reliability, scalability, and correctness of Kafka clusters under production-like workloads.

    Key areas covered by these tests include:

    • Dynamic Configuration: Updates to configurations without manual intervention.
    • Listeners: Configuration and validation of various listener types (Internal, External, LoadBalancer, NodePort, Route).
    • Node Pool Management: Managing broker IDs and scaling via node pools.
    • Lifecycle & Upgrades: Version upgrades and rolling updates.
    • Resource Management: Quotas, tiered storage, and volume management (including JBOD and resizing).
    • Observability: Metrics for Kafka, Cruise Control, Kafka Connect, and the Cluster Operator.
    • Security: TLS, certificate management, and authentication (Scram-SHA, etc.).

  5. Overview of the systemtest package structure

    main

    The systemtest package is organized into main (support classes) and test.

    Notable Modules in main:

    • kafkaclients: Client implementations used in tests.
    • matchers: Implementation for checking Cluster Operator logs.
    • utils: Shared utility methods for common test actions.
    • resources: CRUD methods for managing the lifecycle of Strimzi, Kafka, Kafka Connect, Kafka Bridge, and Kafka Mirror Maker resources.
    • templates: Predefined templates for resource creation (e.g., resource.createResource(extensionContext, template.build())).

    Notable Classes:

    • Environment: A singleton that loads test environment variables.
    • TestConstants: Interface for test constants.
    • SetupClusterOperator: Encapsulates Cluster Operator installation (Olm, Helm, or Bundle). Provides rollbackToDefaultConfiguration() and defaultInstallation() (returns SetupClusterOperatorBuilder).
    • TestStorage: Generates and stores values in a specific ExtensionContext using a ConcurrentHashMap.
    • TestSuiteNamespaceManager: Manages additional namespaces required for a test suite before @BeforeAll execution.

  6. Overview of the HTTP Bridge interface

    main

    The HTTP Bridge provides a RESTful interface that allows HTTP-based clients to interact with a Kafka cluster using HTTP requests instead of the native Kafka protocol. This is useful for clients that cannot or should not use the native Kafka protocol.

    The API exposes two main resources via endpoints:

    • consumers: To manage consumer lifecycles and offsets.
    • topics: To interact with producers and topic metadata.

    Note that these resources relate specifically to the HTTP Bridge's management of consumers and producers, not to the native Kafka consumers and producers connected directly to the cluster. The bridge supports JSON or binary message formats and uses standard HTTP response codes for error handling.

  7. Overview of Strimzi Drain Cleaner

    main

    Strimzi Drain Cleaner is a utility designed to assist in moving Kafka pods deployed by Strimzi from Kubernetes nodes that are undergoing a drain process.

    By using this utility, you allow the Strimzi operator to manage the movement of pods instead of relying on standard Kubernetes node draining. This ensures that Kafka topics do not become under-replicated during the node draining process, maintaining cluster health and data availability.

  8. Understand Kafka producers, consumers, and consumer groups

    main

    In Apache Kafka, data movement is handled by producers and consumers interacting with brokers via topics.

    • Producers: Send messages to a broker topic. Messages are written to the end offset of a partition. Distribution is typically handled via round-robin or by using a specific message key to target a partition.
    • Consumers: Subscribe to a topic and read messages based on the topic, partition, and offset.
    • Consumer Groups: A mechanism to scale data processing. By assigning a group.id, multiple consumers can share the load of a single topic. Within a group, each partition is typically assigned to only one consumer, allowing the data stream to be spread across all members of the group.
    • Offsets: A unique identifier representing a message's position within a partition. Consumers use offsets to track their progress.
    • Committed Offsets: To ensure continuity, consumers write their progress to an offset commit log. This information is stored in a special internal topic named __consumer_offsets, which tracks the last and next offset for each consumer group.

  10. Manage KafkaUser resources with the User Operator

    main

    The User Operator is responsible for managing KafkaUser resources within a Kafka cluster. It handles user authentication, authorization, and lifecycle operations.

    Key capabilities include:

    • Authentication Mechanisms: Support for TLS, SCRAM-SHA-512, and external TLS.
    • Authorization: Managing access control via ACLs.
    • Quota Enforcement: Applying quotas to specific users.
    • Secret Management: Managing secrets with custom prefixes.
    • Lifecycle Operations: Creating, updating, and deleting users to ensure reliable management.

  11. Understand TLS communication components in KRaft mode

    main

    Strimzi uses TLS to encrypt communication between various Kafka components when running in KRaft mode. This includes:

    • Kafka brokers and controllers
    • Strimzi operators and Kafka
    • Cruise Control and Kafka brokers
    • Kafka Exporter and Kafka brokers
    • Clients (producers/consumers) and Kafka brokers

    Client-to-broker encryption is controlled via the tls property within the listener configuration of the Kafka custom resource.

  12. Tools for Kafka tuning

    main

    Use the following tools and features to assist with Kafka cluster optimization and management:

    • Cruise Control: Generates optimization proposals to help assess and implement cluster rebalancing.
    • Strimzi Quotas plugin: Sets limits on brokers to manage resource usage.
    • Rack configuration: Spreads broker partitions across different racks to ensure high availability and allows consumers to fetch data from the nearest replica, reducing latency and network costs.