Lab: Kafka Cluster¶
Pairs with: Kafka Deep Dive
A 3-broker Kafka cluster in KRaft mode (no ZooKeeper). Create a real topic, kill a real broker mid-traffic, and watch a real leader election and consumer-group rebalance — not a canvas animation.
docker-compose.yml¶
name: labs-kafka
x-kafka-common: &kafka-common
image: apache/kafka:3.8.0
environment: &kafka-common-env
KAFKA_PROCESS_ROLES: broker,controller
KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka1:9093,2@kafka2:9093,3@kafka3:9093
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 3
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 2
KAFKA_AUTO_CREATE_TOPICS_ENABLE: "false"
# Fixed cluster ID (base64 UUID) so all three brokers agree on identity across restarts.
CLUSTER_ID: ZmxpZ2h0LWNoZWNrLTEyMzQ1
services:
kafka1:
<<: *kafka-common
hostname: kafka1
environment:
<<: *kafka-common-env
KAFKA_NODE_ID: 1
KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka1:9092
ports:
- "19092:9092"
healthcheck:
test: ["CMD-SHELL", "/opt/kafka/bin/kafka-broker-api-versions.sh --bootstrap-server localhost:9092 || exit 1"]
interval: 5s
timeout: 5s
retries: 20
kafka2:
<<: *kafka-common
hostname: kafka2
environment:
<<: *kafka-common-env
KAFKA_NODE_ID: 2
KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka2:9092
ports:
- "29092:9092"
healthcheck:
test: ["CMD-SHELL", "/opt/kafka/bin/kafka-broker-api-versions.sh --bootstrap-server localhost:9092 || exit 1"]
interval: 5s
timeout: 5s
retries: 20
kafka3:
<<: *kafka-common
hostname: kafka3
environment:
<<: *kafka-common-env
KAFKA_NODE_ID: 3
KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka3:9092
ports:
- "39092:9092"
healthcheck:
test: ["CMD-SHELL", "/opt/kafka/bin/kafka-broker-api-versions.sh --bootstrap-server localhost:9092 || exit 1"]
interval: 5s
timeout: 5s
retries: 20
kafka-ui:
image: provectuslabs/kafka-ui:latest
depends_on:
- kafka1
- kafka2
- kafka3
environment:
KAFKA_CLUSTERS_0_NAME: local
KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka1:9092,kafka2:9092,kafka3:9092
ports:
- "8080:8080"
Exercises¶
The full step-by-step walkthrough (create a topic, kill a broker and predict what happens to leadership/ISR, watch a live consumer-group rebalance, check consumer lag) lives in the lab's README: