DOCSEARCHbyAlgolia
Submit Docs

Apache Kafka

repository·trunk·Indexed 12 days ago

https://github.com/apache/kafka

A distributed event streaming platform for high-performance data pipelines, streaming analytics, and mission-critical applications. Documentation covers the Kafka message protocol, JSON definitions for requests and responses, MessageData objects, and the Kafka API-checker plugins (KIP-1265) for validating API usage.

Tokens
242.5K
Snippets
500
Records
876
Agent score
98%

What's inside Kafka

  1. Overview of Kafka Core APIs

    trunk

    Kafka provides six core APIs for different interaction patterns with the cluster:

    1. Producer API: For sending streams of data to topics.
    2. Consumer API: For reading streams of data from topics.
    3. Share Consumer API: For cooperative data consumption within a shared group.
    4. Streams API: For transforming data streams from input topics to output topics.
    5. Connect API: For implementing connectors that pull data from source systems into Kafka or push data from Kafka into sink systems.
    6. Admin API: For managing and inspecting Kafka objects like topics, brokers, and ACLs.

    Note: While Kafka uses a language-independent protocol, only the Java clients are maintained as part of the main Kafka project. Other language clients are maintained as independent open-source projects.

  2. Overview of Kafka API-checker plugins

    trunk

    The Kafka API-checker plugins (KIP-1265) are used to validate API usage. They are built as a separate Gradle build that is included in the main Kafka build via pluginManagement { includeBuild 'api-checker' }. This separation keeps Maven runtime dependencies off the main Kafka build classpath.

    Subprojects and Artifacts

    SubprojectPublished ArtifactAudience
    :coreorg.apache.kafka:kafka-api-checker-coreShared scanner, validator, and reporter (uses ASM). Required by both plugin jars.
    :gradle-pluginsorg.apache.kafka:kafka-internal-api-checker-gradle-plugin + plugin markers for org.apache.kafka.public-api-checker and org.apache.kafka.internal-api-checkerThe Kafka-internal producer-side checker and the consumer-side Gradle checker.
    :maven-pluginorg.apache.kafka:kafka-internal-api-checker-maven-pluginMaven equivalent of the consumer-side checker.

    For end-user usage (Gradle/Maven snippets, @SuppressKafkaInternalApiUsage, and audience-inheritance rules), refer to docs/apis/internal-api-checker.md.

  3. What is Apache Kafka and how does it work?

    trunk

    Apache Kafka is a distributed event streaming platform designed for high-performance, scalable, and fault-tolerant data handling. It functions as a system of servers and clients communicating via a TCP network protocol.

    Core Capabilities

    1. Publish and Subscribe: Write (publish) and read (subscribe to) streams of events, including importing/exporting data from other systems.
    2. Store: Durably and reliably store event streams for any desired duration.
    3. Process: Process event streams either in real-time as they occur or retrospectively.

    System Components

    • Servers (Brokers): Kafka runs as a cluster of servers. Some servers act as the storage layer (brokers), while others run Kafka Connect to integrate with external systems like relational databases.
    • Clients: Distributed applications or microservices that read, write, and process event streams. Clients are available for Java, Scala, Go, Python, C/C++, and via REST APIs.

  4. What is Kafka Connect

    trunk

    Kafka Connect is a framework designed for scalable and reliable data streaming between Apache Kafka and external systems. It allows you to define connectors to move large collections of data into Kafka (ingestion) or out of Kafka (export).

    Key use cases include:

    • Ingestion: Moving data from databases or application metrics into Kafka topics for low-latency stream processing.
    • Export: Delivering data from Kafka topics into secondary storage, query systems, or batch systems for offline analysis.

    Core features include:

    • Common Framework: Standardizes integration, simplifying development and management.
    • Deployment Modes: Supports both Distributed (for large, centrally managed organizational services) and Standalone (for development, testing, or small deployments) modes.
    • REST Interface: Allows submitting and managing connectors via a REST API.
    • Automatic Offset Management: Handles the offset commit process automatically, reducing complexity for connector developers.
    • Scalability: Built on Kafka's group management protocol, allowing you to scale by adding more workers to a cluster.

  5. Overview of the Kafka Streams DSL

    trunk

    The Kafka Streams DSL (Domain Specific Language) is a high-level, declarative, and functional programming API built on top of the Streams Processor API. It is recommended for most users because it allows expressing complex data processing logic—such as aggregations, joins, and windowing—in just a few lines of code.

    To build an application using the DSL, follow these steps:

    1. Specify input streams: Read one or more input streams from Kafka topics.
    2. Compose transformations: Apply transformations (stateless or stateful) to these streams.
    3. Write output: Write the resulting streams back to Kafka topics or expose them via Interactive Queries.

  6. What is Kafka Streams?

    trunk

    Kafka Streams is a lightweight Java client library used for processing and analyzing data stored in Apache Kafka. It is designed to be embedded within any Java application, requiring no external dependencies other than Kafka itself.

    Key features include:

    • Scalability: Uses Kafka's partitioning model to scale horizontally by running multiple instances of the same application.
    • Fault-Tolerance: Supports fault-tolerant local state for efficient stateful operations like windowed joins and aggregations.
    • Processing Guarantees: Supports exactly-once processing semantics.
    • Low Latency: Employs one-record-at-a-time processing to achieve millisecond latency.
    • Flexible APIs: Provides a high-level Streams DSL for common transformations and a low-level Processor API for custom logic.

  7. Explore the Kafka Ecosystem

    trunk

    Beyond the core Apache Kafka distribution, there is a wide range of tools and integrations available for various use cases. These include:

    • Stream Processing Systems: Frameworks designed to process data streams in real-time.
    • Hadoop Integration: Tools for connecting Kafka with the Hadoop ecosystem for data storage and processing.
    • Monitoring: Solutions for observing Kafka cluster health, performance, and metrics.
    • Deployment Tools: Utilities for managing and automating the deployment of Kafka infrastructure.

    For a comprehensive list of these tools, refer to the official Kafka Ecosystem page.

  8. Kafka Design Motivation and Core Goals

    trunk

    Kafka is designed as a unified platform for real-time data feeds, functioning more like a distributed database log than a traditional messaging system. Its architecture is driven by several key requirements:

    • High Throughput: Capable of handling high-volume event streams, such as real-time log aggregation.
    • Large Data Backlogs: Designed to handle massive data loads from offline systems gracefully.
    • Low Latency: Supports traditional messaging use-cases requiring rapid delivery.
    • Distributed Processing: Supports partitioned, distributed, real-time processing to create derived data feeds.
    • Fault Tolerance: Guarantees data integrity and availability in the presence of machine failures.

  9. Notable changes in Kafka 4.4.0

    trunk

    Kafka 4.4.0 introduces several breaking changes, deprecations, and new features. Key updates include:

    Deprecations (Scheduled for removal in 5.0)

    • ClientQuotaCallback#updateClusterMetadata: Custom implementations no longer need to override this; a default no-op is provided.
    • Group Coordinator Metrics: Yammer-based metrics are deprecated. Use the new kafka.server:type=group-coordinator-metrics namespace instead.
    • broker.id: This configuration is deprecated. Use node.id instead. Tiered storage plugins must also transition to using node.id.

    Breaking Changes & Compatibility

    • Storage Directory Compatibility: Directories formatted by the kafka-storage tool are not forward-compatible. The Kafka broker version must be the same as or newer than the version of the kafka-storage tool used for formatting.
    • OAUTHBEARER JWT Validation: If sasl.oauthbearer.jwks.endpoint.url is configured, the broker will now fail to start if sasl.oauthbearer.expected.audience or sasl.oauthbearer.expected.issuer are not set. To bypass this, set sasl.oauthbearer.allow.unverified.audience or sasl.oauthbearer.allow.unverified.issuer to true.

    New Features & Tools

    • Tiered Storage Redundancy: New configs remote.copy.lag.bytes and remote.copy.lag.ms allow for delayed uploads to reduce redundancy.
    • Share Groups DLQ: Supports Dead Letter Queues when share.version is upgraded to 2. Configure via errors.deadletterqueue.topic.name.
    • Kafka Connect: New internal.topics.automatic.creation.enable config allows controlling the automatic creation of internal topics.
    • CLI Updates: kafka-cluster.sh api-versions is the new way to check supported API versions; kafka-broker-api-versions.sh is deprecated.

  10. Use the GraalVM Based Native Apache Kafka Docker Image

    trunk

    The Native Apache Kafka Docker Image allows you to launch brokers with sub-second startup times and a minimal memory footprint by using a native Kafka executable. This executable is created via ahead-of-time (AOT) compilation using the GraalVM native-image tool.

    Warning: This image is experimental and intended for local development and testing purposes only. It is not recommended for production use.

    This feature was introduced with KIP-974.

  11. Supported Kafka security measures

    trunk

    Kafka provides four primary security layers that can be configured independently or in combination:

    1. Authentication: Verifies the identity of clients (producers/consumers), brokers, and tools. Supported via SSL or SASL mechanisms:
      • SASL/GSSAPI (Kerberos)
      • SASL/PLAIN
      • SASL/SCRAM-SHA-256 and SASL/SCRAM-SHA-512
      • SASL/OAUTHBEARER
    2. Encryption: Protects data in transit between brokers, clients, and tools using SSL. Note that enabling SSL may cause performance degradation depending on CPU and JVM implementation.
    3. Authorization: Controls read and write operations by clients.
    4. Pluggable Authorization: Allows integration with external authorization services.

    Security is optional; Kafka supports non-secured clusters and mixed environments containing both authenticated/unauthenticated and encrypted/non-encrypted clients.

  12. Understand Geo-Replication with MirrorMaker

    trunk

    Kafka administrators can use MirrorMaker (version 2) to replicate data across different Kafka clusters, data centers, or geo-regions. MirrorMaker is built on the Kafka Connect framework and is used for scenarios such as disaster recovery, cloud migration, or feeding edge clusters into a central aggregate cluster.

    Key capabilities include:

    • Topic Replication: Replicates both data and topic configurations.
    • Consumer Group Migration: Replicates consumer groups and their offsets to allow applications to migrate between clusters.
    • ACL Replication: Replicates Access Control Lists.
    • Partition Preservation: Maintains partitioning during replication.
    • Automatic Discovery: Automatically detects new topics and partitions.
    • Scalability: Provides fault-tolerant and horizontally scalable operations with metrics for end-to-end replication latency.

    Note: MirrorMaker performs inter-cluster replication, which is distinct from Kafka's standard intra-cluster replication.