Kafka Internals & Pulsar Comparison¶

Prerequisites: Kafka consumer groups, Distributed Systems

← Messaging Overview

Why This Exists¶

You know Kafka works. But when you're designing a system, you need to know:

1. How does Kafka actually guarantee order and durability? (answer: in-sync replicas + leader)
2. What breaks during a rebalance? (answer: eager rebalances stop group consumption; cooperative rebalancing reduces that disruption; latency still spikes for moved partitions)
3. Can you get exactly-once semantics? (answer: yes, but read carefully)
4. What is consumer lag, and why does it matter? (answer: it's the outage metric)
5. When is Kafka wrong for this job? (answer: when you need multi-tenancy, or Pulsar's scaling model)

This page teaches the internals, the failure modes, and the Pulsar alternative.

Part 1: Kafka Internals — How It Actually Works¶

In-Sync Replicas (ISR): The Source of Truth¶

Every partition has N replicas distributed across N brokers. One is the leader; the rest are followers. The ISR is the subset of replicas that are fully caught up: ISR ⊆ replicas. A lagging follower is still a replica; it is not in the ISR until it catches up.

Topic "orders", Partition 0, RF=3

Leader (Broker 0)              Follower (Broker 1)         Follower (Broker 2)
┌──────────────────────┐      ┌──────────────────────┐     ┌──────────────────────┐
│ Offset 0..100        │      │ Offset 0..100        │     │ Offset 0..95         │
│ (all committed)      │      │ (in ISR)             │     │ (lagging → NOT ISR)  │
│                      │      │                      │     │                      │
│ Producer writes:     │  ──→ │ Leader replicates    │ ──→ │ Followers replicate  │
│ [msg101]             │      │ [msg101]             │     │ (behind)             │
└──────────────────────┘      └──────────────────────┘     └──────────────────────┘
         ↑
   Consumers default to the leader (KIP-392: can fetch from closest replica)

The ISR guarantee: A message is considered committed (durable) only after: - The leader writes it - Every current ISR member acknowledges it (acks=all)

If the leader crashes before a replica syncs, that message is lost if an out-of-ISR replica is elected (unclean election).

acks=all vs min.insync.replicas: acks=all waits for every current ISR member, not for min.insync.replicas replicas. min.insync.replicas is a floor: if |ISR| < min.insync.replicas, an acks=all produce fails with NotEnoughReplicas. With acks=all and min.insync.replicas=1, if ISR has 3 members, all 3 still ack. acks=1 (WaitForLocal) is independent of min.insync.replicas — the leader acks locally even when ISR is larger.

Durability recipe: acks=all + min.insync.replicas=2 + RF=3 + unclean.leader.election.enable=false.

// Kafka Producer config
config.Producer.RequiredAcks = sarama.WaitForAll  // acks=all: wait for every current ISR member
// min.insync.replicas is a floor, not the ack count. If |ISR| < min.insync.replicas → NotEnoughReplicas

// vs
config.Producer.RequiredAcks = sarama.WaitForLocal  // acks=1: leader only; independent of min.insync.replicas
// Faster; leader crash before replication can lose the write

Rebalancing: The Latency Killer¶

When a consumer joins/leaves the group, Kafka reassigns partitions. Eager (stop-the-world) rebalances pause all consumption for the group; that window is often seconds, not tens of milliseconds. Cooperative sticky (Kafka 2.4+) only pauses partitions that actually move.

This is not the same as adding a broker. Adding a broker triggers replica reassignment (data movement). It does not by itself cause a consumer-group rebalance.

Scenario: 3 consumers, 6 partitions (eager rebalance)

State 0 (steady)          State 1 (C3 joins)            State 2 (rebalance done)
─────────────────         ──────────────────            ─────────────────────
C1: P0, P1                C1, C2, C3 all STOP          C1: P0, P2
C2: P2, P3                No consumption               C2: P1, P4
C3: P4, P5                (often seconds; eager)       C3: P3, P5

Eager timeline (order of seconds, not 25ms):
  JoinGroup → all members revoke partitions → stop fetching
  group leader computes assignment → SyncGroup
  members resume on new assignment

Cooperative sticky: unaffected partitions keep fetching; only moving
partitions pause. Prefer this over eager for latency-sensitive groups.

Why rebalancing is slow:

1. Pause/resume is serial — cannot parallelize
2. Offset commit must succeed — if commit fails, consumer is stuck
3. State machine in each consumer — must transition through JOIN_WAIT, OFFSET_COMMIT, RUNNING

Failure scenario:

C1 crashes mid-rebalance
├─ Was committing offset
├─ Commit times out
├─ Whole group stuck in limbo
├─ After session timeout (30-45 sec)
├─ Rebalance retries
└─ Lag spike

How to minimize: - max.poll.interval.ms: How long you can take per poll before a rebalance trigger (default 300s). Increase if processing is slow. - session.timeout.ms: How long before a consumer is declared dead (default 45s). Lower = faster failover, but more false positives and rebalances. Must sit in [heartbeat.interval.ms, group.max.session.timeout.ms].

Consumer Lag: The Real Health Metric¶

Definition: lag = latest_offset_in_partition - consumer_committed_offset

Partition 0:
  Latest offset: 1000
  Consumer committed: 950
  Lag = 50 messages

Meaning: Consumer is 50 messages behind production.

Why it matters:

• Lag = 0: System is keeping up
• Lag = 100: System is ~1 second behind (if 100 msg/sec production)
• Lag = 10,000: System is ~100 seconds behind; data is stale

SLO violation threshold: If SLO says "data lag < 60 seconds" and you see lag > 600 messages (assuming 10 msg/sec), you have violated SLO and should page.

Causes of growing lag: 1. Consumer is slow (long message processing time) 2. Consumer crashed (backlog accumulating) 3. Rebalance (lag builds during rebalance window) 4. Network issue (consumer can't fetch) 5. GC pause (Java: 500ms GC = lag increase = 500ms × producer_rate)

Monitoring lag:

from kafka import KafkaConsumer, TopicPartition

consumer = KafkaConsumer(
    bootstrap_servers=["localhost:9092"],
    group_id="order-processor",
)
partitions = [
    TopicPartition("orders", p)
    for p in consumer.partitions_for_topic("orders") or []
]
end_offsets = consumer.end_offsets(partitions)
for tp in partitions:
    committed = consumer.committed(tp) or 0
    print(f"P{tp.partition}: lag={end_offsets[tp] - committed}")

Ordering Guarantees¶

Topic "orders" with 3 partitions

P0: order 1, order 4, order 7
P1: order 2, order 5, order 8
P2: order 3, order 6, order 9

Consumer sees: [1, 2, 3, 4, 5, 6, 7, 8, 9] (if polling all partitions)
            or [1, 4, 7, 2, 5, 8, 3, 6, 9] (if fetching in round-robin)
            or any interleaving

Only the Kafka log within a partition is ordered. Consumer processing and downstream side effects are not automatic log-order guarantees.

How to get a global Kafka log order: a single partition. That kills parallelism (max throughput = one partition). An ordering key only colocates related records on one partition — per-key / per-partition log order, not global order, and still not a processing or downstream guarantee.

// Kafka producer routing by key
producer.SendMessage(&sarama.ProducerMessage{
    Topic: "orders",
    Key:   sarama.StringEncoder("order_123"),  // Same key → same partition
    Value: sarama.StringEncoder(orderJSON),
})

Exactly-Once Semantics (Read Carefully)¶

Split two claims that are often mashed together:

• Broker / Kafka-stream EOS (0.11+): idempotent producer + transactions can give exactly-once within Kafka (produce + offset commit in one txn; downstream consumers with isolation.level=read_committed skip aborted txns).
• End-to-end EOS (source → Kafka → your DB → sink): not a generic database transaction. A Python with transaction: db.insert(); kafka_consumer.commit() is not Kafka EOS.

# Kafka EOS: enable.idempotence + transactional producer + sendOffsetsToTransaction
# Offsets and output records commit together *in Kafka*. Your SQL DB is not in this txn.

producer = KafkaProducer(
    bootstrap_servers=["localhost:9092"],
    enable_idempotence=True,
    transactional_id="order-processor-1",
    acks="all",
)
producer.init_transactions()

consumer = KafkaConsumer(
    "orders",
    bootstrap_servers=["localhost:9092"],
    group_id="order-processor",
    enable_auto_commit=False,
    isolation_level="read_committed",
)

for msg in consumer:
    producer.begin_transaction()
    try:
        producer.send("orders-processed", process(msg))
        producer.send_offsets_to_transaction(
            {TopicPartition(msg.topic, msg.partition): OffsetAndMetadata(msg.offset + 1, "")},
            consumer.config["group_id"],
        )
        producer.commit_transaction()
    except Exception:
        producer.abort_transaction()
        raise

A side-effecting idempotent sink (upsert by business key) is still required if you write outside Kafka. In practice: most systems use at-least-once + idempotent handlers unless they stay inside Kafka transactions.

# At-least-once + idempotent
def at_least_once_consumer():
    seen = set()  # or cache like Redis
    for msg in kafka_consumer:
        if msg.id in seen:
            continue  # Skip duplicate
        process(msg)
        db.insert_dedupe_key(msg.id)  # Idempotent key in DB
        seen.add(msg.id)
        kafka_consumer.commit_async()

Replication Lag (Not Consumer Lag)¶

This is a different problem: followers lag behind the leader.

Leader has offset 0..100
Follower1 has offset 0..95 (lagging)
Follower2 has offset 0..100 (in sync)

acks=all waits for every *current* ISR member, not for min.insync.replicas.
If follower1 is still in ISR, the produce waits on it (slow).
If follower1 falls too far behind (replica.lag.time.max.ms, default 30s),
it is kicked out of ISR — then acks=all no longer waits on it.
If that shrinks |ISR| below min.insync.replicas, new acks=all produces fail.

Causes: - Follower is slow (overloaded) - Network hiccup between leader and follower - Follower rebooting

Monitoring: Check kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions

Part 2: Pulsar — Kafka's Alternative¶

Apache Pulsar is a newer messaging system (Splunk/Yahoo backed) that solves some of Kafka's pain points. It's not "better," but it's different.

Mental Model: Pulsar's Architecture¶

Kafka: All broker nodes are peers
       Each broker stores partition replicas

Pulsar: Tiered architecture
        ├─ Brokers (stateless) ← can add/remove without rebalancing
        ├─ BookKeepers (stateful durable log) ← separate from brokers
        └─ Metadata store (ZK/etcd) ← coordination

Key Differences¶

Pulsar Partitioning Model¶

Kafka: Topic → Partitions → Replicas on Brokers

Pulsar: Tenant → Namespace → Topic → Partitions → Ledgers on BookKeepers

Tenant: "company-a"
├─ Namespace: "production"
│  └─ Topic: "orders"
│     ├─ Partition-0: [Ledger1 on BK1, Ledger2 on BK2, Ledger3 on BK3]
│     └─ Partition-1: [Ledger4 on BK1, Ledger5 on BK2, Ledger6 on BK3]
└─ Namespace: "staging"
   └─ Topic: "orders"
      └─ ...

Ledgers: Immutable log segments. When full, Pulsar creates a new ledger. BookKeepers store them. This separation means: - Storage can scale independent of brokers - Brokers are stateless (restart = no rebalancing) - Old ledgers can be offloaded to S3 (Pulsar feature)

Multi-tenancy in Pulsar¶

Kafka cluster: One team or many teams sharing
              → One team misconfigures topic → affects everyone
              → One team's consumer lag spike → rebalance affects everyone

Pulsar cluster: Tenants are isolated
               → company-a:prod isolated from company-b:prod
               → Quotas per tenant (message rate, storage)
               → Different policies (retention, replication)

Example: Stripe could give each customer a Pulsar tenant with isolated quotas.

Geo-Replication¶

Kafka: Manually set up MirrorMaker or Confluent Replicator. Complex.

Pulsar: Built-in.

# Pulsar: One-line configuration
pulsar-admin topics create-partitioned-topic \
  persistent://tenant/namespace/topic \
  --replication-clusters us-east,us-west,eu

# Now topic is automatically replicated across regions

Consumer Subscriptions¶

Kafka has consumer groups. Pulsar has subscriptions with multiple modes:

Advantage: No rebalancing on Exclusive/Failover. When active consumer crashes, a standby instantly takes over (no rebalance delay).

Tiered Storage¶

Pulsar can automatically offload old ledgers to S3/GCS.

BookKeeper storage:            S3:
├─ Recent ledgers              ├─ Old ledgers
│ (hot, in memory)             │ (cold, archived)
│ ~10GB                        │ ~10TB
│ Fast reads                   │ Cheap, slow reads
└─ Offload threshold:          └─ Can be replayed if needed
  when ledger is 2 weeks old

Why: Kafka requires brokers to store all data on local disk. Pulsar lets you keep only recent data "hot" and archive old data cheaply.

Exactly-Once Deduplication¶

Pulsar has built-in deduplication on a producer sequence ID:

# Pulsar: Automatic deduplication
producer = client.create_producer(
    topic='my-topic',
    deduplication_enabled=True,  # Global dedup, window = 1 hour
)

msg = producer.send(
    content='message',
    sequence_id=123  # Producer seq ID
)

# If producer crashes and retries:
# - Message with seq_id=123 is idempotent
# - Sent 100 times → only stored once

Kafka requires you to implement dedup yourself.

Kafka vs Pulsar: Decision Matrix¶

Real Numbers: When Pulsar Wins¶

Scenario: 500,000 RPS, multi-region (US + EU), 5 teams

Kafka: - One cluster per region (2 clusters) - Each cluster: ~15 brokers - Each team's topic: ~6 partitions - Cost: Brokers at full RF=3 size - Rebalancing overhead: When any broker restarts, 10-30s of elevated latency cluster-wide - Multi-region: Manual federation, lag = 5-60 seconds

Pulsar: - One cluster, multi-region replication enabled - BookKeepers: ~12 (scales independently) - Brokers: ~6 (mostly idle) - Each team: 3 partitions - Cost: Lower (BookKeepers cheaper than Kafka brokers) - Rebalancing: None (brokers are stateless) - Multi-region: Automatic, lag < 100ms

Cost difference: Kafka = $20k/month, Pulsar = $8k/month (rough estimate at this scale)

Antipatterns¶

Antipattern 1: Partition Count Too Low¶

Topic: 10 partitions
Consumers: 100

→ 90 consumers are idle
→ Parallelism capped at 10
→ Max throughput = single consumer speed × 10

If single consumer can handle 10k msg/s:
Max throughput = 100k msg/s
Actual throughput = 100k msg/s (OK)

But if production is 200k msg/s:
Lag builds forever

Fix: Over-provision partitions early. 10x your expected consumer parallelism.

Antipattern 2: Not Monitoring Consumer Lag¶

System appears healthy for weeks
Behind the scenes: lag is growing (consumer is slow)
Customer complains: "My data is 8 hours old"
Postmortem: Nobody was monitoring lag

Fix: Alert on lag > SLO threshold. Lag is the real health metric.

Antipattern 3: min.insync.replicas=1¶

Default setting: only leader needs to ack
Leader crashes: message written but not replicated
New leader elected: message is gone forever
Customer notices: data loss

Fix: Set min.insync.replicas=2 (or higher). Producers wait for replicas.

Antipattern 4: No Backpressure on Producer¶

Producer: fire and forget
Queue starts filling (consumer slow)
Queue fills: 1000, 10000, 100000 messages
Memory: producer process OOM
System: cascading failure

Better: Producer should respect queue length
If queue > threshold: producer backed off or rejected

Fix: Use callbacks/futures in producer; track inflight requests.

Interview Questions¶

Key Takeaways¶

Previous: Kafka consumer groups | Next: Queue patterns