Kafka Deep Dive¶

Prerequisites: Message Queue Patterns, Basic distributed systems

Why This Exists¶

Start with a simple problem: 1 producer → 1 topic → 1 consumer, 10 MB/s.

Works perfectly. Now: what happens at 500 MB/s?

A single consumer can't keep up. You need to parallelize consumption. But you can't just add random consumers — you'd process the same messages multiple times or miss messages.

This is why Kafka has partitions and consumer groups.

Mental Model¶

Topic "orders" with 3 partitions:

Producer ──┬──> Partition 0: [msg1, msg4, msg7, msg10...]
           ├──> Partition 1: [msg2, msg5, msg8, msg11...]
           └──> Partition 2: [msg3, msg6, msg9, msg12...]

Consumer Group "order-processor":
  Consumer C0 ──> Partition 0
  Consumer C1 ──> Partition 1
  Consumer C2 ──> Partition 2

Key rules: 1. Each partition is consumed by exactly one consumer within a group at a time 2. The Kafka log within a partition is an ordered append sequence 3. Messages across partitions have no global order 4. Consumer processing and end-to-end side effects are not automatic log-order guarantees 5. More consumers than partitions → some consumers are idle

Abstraction Levels¶

See Exactly-Once Semantics and Ordering Guarantees below for the full detail behind each claim above.

Architecture¶

graph LR
    subgraph Producers
        P1[Producer 1]
        P2[Producer 2]
    end

    subgraph Topic["Topic: orders (3 partitions, RF=3)"]
        subgraph Broker1["Broker 1 (Leader P0)"]
            PA["P0: [0,1,2,3...]"]
        end
        subgraph Broker2["Broker 2 (Leader P1)"]
            PB["P1: [0,1,2,3...]"]
        end
        subgraph Broker3["Broker 3 (Leader P2)"]
            PC["P2: [0,1,2,3...]"]
        end
    end

    subgraph CG["Consumer Group: order-processor"]
        C0["C0\nOffset: 42"]
        C1["C1\nOffset: 38"]
        C2["C2\nOffset: 45"]
    end

    P1 -->|key hash| PA
    P1 -->|key hash| PB
    P2 -->|key hash| PC
    PA --> C0
    PB --> C1
    PC --> C2

Interactive Kafka Simulation¶

Try: 1. Start producer → observe lag stays near 0 when consumers keep up 2. Kill a consumer → watch lag build up, observe rebalancing 3. Add a partition → can increase throughput but note the rebalancing cost

How It Works Internally¶

Offsets¶

Every message in a partition has a monotonically increasing offset. Kafka's committed-offset convention is the offset of the next record the consumer should read, not the offset of the last one it processed — so after successfully processing the message at offset 5, the consumer commits 6, and a restart resumes from 6, not 5 (committing 5 would mean re-reading and reprocessing that same message on restart).

Partition 0: offset 0, 1, 2, 3, 4, 5, 6...
                                     ↑
                        Consumer just processed offset 5
                        → commits 6 (next record to consume)
                        → on restart, resume from 6

Auto-commit pitfall: Kafka can auto-commit the offset before the message is actually processed. If the consumer crashes between auto-commit and processing → message loss.

Manual commit after processing → at-least-once semantics (safe: possible duplicates, no losses).

Rebalancing¶

When a consumer joins or leaves a group, Kafka triggers a rebalance and the group coordinator redistributes partitions.

Eager rebalances stop group consumption while assignments are revoked and redistributed (stop-the-world for the group). Cooperative/incremental rebalancing (Kafka 2.4+) reduces disruption by retaining unaffected assignments and moving partitions incrementally — consumers keep consuming partitions they still own.

Consumer Lag¶

Consumer lag = (Latest offset in partition) − (Committed offset of consumer)

Partition 0: latest offset = 1,000,000
Consumer C0 committed offset = 999,500
Lag = 500 messages

High lag indicates: - Consumer is too slow (processing bottleneck) - Consumer is down - Traffic spike outpacing consumption rate

ISR (In-Sync Replicas)¶

Each partition has one leader and N-1 followers (replicas). The ISR is the set of replicas fully caught up with the leader.

• acks=all (strongest): producer waits for every current ISR member to acknowledge — not min.insync.replicas replicas. min.insync.replicas is a floor: if |ISR| < min.insync.replicas, the produce fails (NotEnoughReplicas)
• acks=1: only the leader acknowledges (independent of min.insync.replicas; risk: leader dies before replication = data loss)
• acks=0: fire and forget (highest throughput, data loss possible)

Consumers default to fetching from the leader. Since KIP-392 they can fetch from the closest replica (replica.selector.class / client.rack) — "always from the leader" is the old default, not a hard rule.

Ordering Guarantees¶

How to keep related messages in the same Kafka log sequence: Use the same partition key. That gives per-partition log order, not an automatic end-to-end processing guarantee.

# All events for user 123 go to the same partition
producer.send(
    topic="user-events",
    key=b"user:123",       # same key → same partition
    value=event_payload
)

Pitfall: If user 123 is extremely active, their partition becomes a hot partition — one consumer processes 10× more messages than others.

Failure Modes¶

Hot Partition¶

• Cause: Highly skewed key distribution (one key generates most messages)
• Symptoms: One partition's lag grows while others are fine; one consumer at 100% CPU
• Detection: Per-partition message rate; consumer lag by partition
• Fix: Add random suffix to hot key (user:123:0, user:123:1), or use null key (round-robin)

Consumer Rebalance Loop¶

• Cause: Consumer takes too long to process → exceeds max.poll.interval.ms → Kafka assumes it's dead → rebalances → same consumer rejoins → repeat
• Symptoms: Frequent rebalances in logs, lag oscillates up and down
• Detection: Group coordinator logs, rebalance frequency metric
• Fix: Increase max.poll.interval.ms OR reduce max.poll.records (process fewer messages per poll) OR optimize consumer processing

Poison Message¶

• Cause: A malformed message causes consumer to crash on every attempt
• Symptoms: Consumer restarts repeatedly, lag on specific partition never decreases
• Detection: Consumer crash logs with same offset repeatedly
• Fix: Dead Letter Queue (DLQ) — after N retries, move to DLQ topic; alerting on DLQ messages

Unclean Leader Election¶

• Cause: All ISR replicas are down; Kafka elects an out-of-sync replica as leader (unclean.leader.election.enable=true)
• Impact: Data loss — messages produced after this replica's last sync are gone
• Fix: Set unclean.leader.election.enable=false for durability-critical topics

Production Debugging¶

Symptom: Consumer lag growing steadily

Diagnostic steps:
1. Check consumer CPU/memory (is processing bottlenecked?)
   → JVM GC pauses? Processing logic slow?
2. Check if consumer is actually consuming
   → Consumer group offset progress over time
   → kafka-consumer-groups.sh --describe
3. Check for rebalances
   → group coordinator logs, rebalance metric
4. Check partition count vs consumer count
   → More partitions than consumers = parallelism bottleneck
5. Check producer throughput increase
   → Topic ingestion rate vs consumer throughput rate
6. Check for poison messages
   → Consumer crash logs, specific partition stuck at same offset

Key metrics:
- consumer_lag_messages (per partition)
- consumer_group_rebalance_count
- broker_network_io, broker_disk_io
- request_handler_avg_idle_percent (< 30% = broker overloaded)
- under_replicated_partitions (> 0 = replication issue)

Scaling Limits¶

• Partition count is permanent — can increase, cannot decrease (without data migration)
• Each partition is a file on disk — too many partitions → too many file handles, slower leader election
• Rule of thumb: < 10,000 partitions per broker
• Consumer lag recovery: scale out consumers (up to partition count)
• Write throughput: scale out by adding partitions and brokers
• Retention: controlled by retention.bytes and retention.ms

Trade-offs¶

Interview Questions¶

Key Takeaways¶