wurstmeister/kafka-docker

To run a Kafka cluster using this project, ensure you have docker-compose installed. Use the following commands to manage the lifecycle of your cluster:

• Start a cluster: docker-compose up -d
• Scale the cluster (e.g., to 3 brokers): docker-compose scale kafka=3
• Stop the cluster: docker-compose stop

Important: By default, each broker will receive a new port number and broker ID on restart. If you require stable IDs and ports, use a specific configuration file like docker-compose-single-broker.yml.

When deploying in Docker Swarm using an overlay network, follow these best practices:

1. Listener Configuration: Use the multiple-listener configuration (separating INSIDE and OUTSIDE traffic) so external clients can connect via the host IP while internal swarm traffic uses the overlay network.
2. Deployment Mode: Use deploy: global in your compose file to ensure exactly one broker per swarm node.
3. Port Mapping: Use compose file version '3.2' or higher and define ports in host mode instead of the default ingress mode to ensure requests are routed to the correct broker.

Example port configuration:

ports:
  - target: 9094
    published: 9094
    protocol: tcp
    mode: host

For modern Kafka setups (0.9.0+), it is recommended to use multiple listener configurations to separate internal and external traffic. This involves setting KAFKA_LISTENERS, KAFKA_ADVERTISED_LISTENERS, KAFKA_LISTENER_SECURITY_PROTOCOL_MAP, and KAFKA_INTER_BROKER_LISTENER_NAME.

Rules:

• No listeners may share a port number.
• An advertised listener must be present by protocol name and port number in the list of listeners.

You can customize Kafka parameters by adding them as environment variables in your docker-compose.yml. The variable name should be prefixed with KAFKA_ followed by the parameter name in uppercase.

Examples:

• Set message.max.bytes: KAFKA_MESSAGE_MAX_BYTES: 2000000
• Disable automatic topic creation: KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'false'

environment:
  KAFKA_MESSAGE_MAX_BYTES: 2000000
  KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'false'

To configure broker rack affinity (e.g., for AWS availability zones), use:

1. KAFKA_BROKER_RACK (explicit value).
2. RACK_COMMAND (a command that returns the rack/zone value).

environment:
  RACK_COMMAND: "curl http://169.254.169.254/latest/meta-data/placement/availability-zone"

The advertised hostname and port can be set explicitly or determined programmatically via commands:

• Hostname: Use KAFKA_ADVERTISED_HOST_NAME (explicit) or HOSTNAME_COMMAND (command).
• Port: Use PORT_COMMAND to determine the port programmatically.

Interpolation: You can inject the result of a command into other KAFKA_XXX variables using the _{COMMAND_NAME} syntax.

environment:
  HOSTNAME_COMMAND: "route -n | awk '/UG[ \t]/{print $$2}'"
  PORT_COMMAND: "docker port $$(hostname) 9092/tcp | cut -d: -f2"
  KAFKA_ADVERTISED_LISTENERS: "PLAINTEXT://_{HOSTNAME_COMMAND}:_{PORT_COMMAND}"

To enable JMX monitoring, configure KAFKA_JMX_OPTS and specify a JMX_PORT. Note that you may need to set java.rmi.server.hostname and com.sun.management.jmxremote.rmi.port to ensure connectivity.

environment:
  KAFKA_JMX_OPTS: "-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=127.0.0.1 -Dcom.sun.management.jmxremote.rmi.port=1099"
  JMX_PORT: 1099

You can set the broker ID using one of two methods:

1. Explicitly: Use the KAFKA_BROKER_ID environment variable.
2. Via Command: Use the BROKER_ID_COMMAND environment variable to run a shell command that returns the ID.

Example using a command: BROKER_ID_COMMAND: "hostname | awk -F'-' '{print $$2}'"

To have Kafka automatically create topics during startup, use the KAFKA_CREATE_TOPICS environment variable. The format is TopicName:Partitions:Replicas[:cleanup_policy]. Multiple topics are separated by a comma (,).

If you need a different delimiter (like a newline), override it using KAFKA_CREATE_TOPICS_SEPARATOR.

environment:
  KAFKA_CREATE_TOPICS: "Topic1:1:3,Topic2:1:1:compact"
  KAFKA_CREATE_TOPICS_SEPARATOR: "$$'\n'"

You can deploy a Kafka broker and a Zookeeper instance using the provided docker-compose.yml. This setup connects Kafka to Zookeeper and exposes the necessary ports for communication.

version: '2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
    restart: unless-stopped

  kafka:
    build: .
    ports:
      - "9092"
    environment:
      DOCKER_API_VERSION: 1.22
      KAFKA_ADVERTISED_HOST_NAME: 192.168.99.100
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    restart: unless-stopped

The Kafka service accepts several environment variables to configure its connection to Zookeeper and its network identity:

• KAFKA_ZOOKEEPER_CONNECT: The connection string for Zookeeper (e.g., zookeeper:2181).
• KAFKA_ADVERTISED_HOST_NAME: The hostname that the Kafka broker will advertise to clients.
• DOCKER_API_VERSION: Specifies the Docker API version to use (e.g., 1.22).