Apache Kafka¶

11:58 PM. Billing's consumer for service-events is nine minutes behind and climbing. The API gateway is still writing tens of thousands of records per second — auth, billing, fraud, search, and the warehouse all read the same stream, each at its own pace. If that stream were a shared database table, the fix would be "wait" or "drop rows." Neither is acceptable tonight.

Before you read on: how do you let five independent consumers read the same firehose at five different speeds, with one of them nine minutes behind, without slowing the producer down and without losing what a faster consumer already deleted?

A request/response API stalls the producer on the slowest consumer. A database table used as a queue lets the second reader see only what the first has not yet deleted. A traditional broker that acknowledges and drops loses the evidence the moment one consumer says "done." Kafka exists because producers and consumers must evolve independently without losing the stream.

What this module covers¶

This module sits on partitioning, data movement, and batch versus stream. Downstream you will use the same topics from Flink.

The central intuition¶

Kafka is not a queue. It is a distributed, replicated, append-only log.

Producers append. Consumers read from an offset. The log stays around for a retention window (or is compacted to the latest value per key). Ten consumer groups can read the same bytes at ten different speeds. Replay is seeking, not begging the producer to send again.

Partition 0 of `service-events`
offset   0     1     2     3     4     5     6     7
         e0 → e1 → e2 → e3 → e4 → e5 → e6 → e7 → (tail)

alert-processor     committed offset 3
warehouse-loader    committed offset 6
fraud-scorer        committed offset 7  (caught up)

A queue that deletes on ack cannot do this. A database table that is not an append log fights the access pattern: sequential write, sequential read, many independent cursors.

Five systems, one event shape¶

Reuse this record throughout the module. Serialise it however you like in labs (JSON is fine); production will argue about Avro and Protobuf later.

{
  "timestamp": "2024-01-15T10:03:45.123Z",
  "customer_id": "cust_1842",
  "user_id": "u_99102",
  "service": "auth",
  "endpoint": "/login",
  "region": "eu-west-1",
  "latency_ms": 87,
  "status_code": 401,
  "bytes": 512
}

The security / observability platform is the running numeric example: ~500k log lines/s, ~2M metric points/s, ~50k security events/s. Multiple processors (alert, enrich, store, train) must not couple back to the fleet of producers.

Where it sits in an architecture¶

flowchart LR
    subgraph producers [Producers]
      API[API / auth / billing]
      Agents[Sidecars / agents]
    end

    subgraph kafka [Kafka cluster]
      T1["topic service-events"]
      T2["topic login-events"]
      T3["compacted user-profile"]
    end

    subgraph consumers [Independent consumer groups]
      Alert[Alerting]
      Flink[Flink jobs]
      WH[Warehouse loader]
      Lake[Lake / Iceberg]
    end

    API --> T1
    API --> T2
    Agents --> T1
    T1 --> Alert
    T1 --> Flink
    T1 --> WH
    T2 --> Flink
    T3 --> Flink
    Flink --> Lake

Producers need three things from this layer: high write throughput, durability they can name (acks, ISR), and a partition key that preserves the ordering they actually care about. Consumers need independent offsets, enough partitions to parallelise, and a retention window longer than their worst outage.

Kafka does not compute "failed logins per user in five minutes". That is Flink. Kafka stores and fans out the raw stream.

What Kafka actually guarantees (and does not)¶

It does not give you:

• Random access by user_id across partitions (scan or materialise elsewhere)
• Cross-partition transactions that include Postgres, Stripe, or a webhook
• Automatic schema compatibility because you "use a registry" (the registry only helps if producers and consumers agree the compatibility mode)

Throughput intuition before you tune¶

A single partition is a single log on a single leader broker, consumed by at most one member of a group. Millions of records per second therefore means many partitions, many leaders, and many consumers — plus the disk sequential-write path that Kafka was built around.

Details live in partitions and replication. The partition simulator is worth five minutes before the labs.

The cluster you are actually operating¶

A Kafka cluster is brokers plus a controller (KRaft in 3.x; ZooKeeper + controller in older deployments). Clients (bootstrap.servers) ask any broker for metadata: which broker leads service-events partition 7, where the ISR is, what the topic configs are.

You will live in a handful of objects:

Producers batch (linger.ms, batch.size) and compress. Followers fetch those batches. Consumers fetch them again. Disk sequential write is the happy path; random reads of old segments (a warehouse backfill) is the unhappy path that evicts page cache from the live tail.

If you remember one operational sentence: Kafka is a disk and metadata system that happens to speak a consumer protocol. Incidents that look like "consumer lag" are often disk, ISR, or rebalance.

Contracts to write down before the first topic¶

Do not create service-events until you can fill this:

key:           customer_id          # order scope
partitions:    24                   # headroom for consumers
RF / min.ISR:  3 / 2
acks:          all
cleanup:       delete, 48h          # or compact for changelogs
consumers:     alert, warehouse, flink-fraud   # separate group ids
poison policy: DLQ (logs) / halt (payments)
schema:        Avro + BACKWARD      # not "JSON, we'll be careful"

E-commerce orders will differ (tighter EOS, maybe fewer partitions for order). IoT device-state will differ (compact). Observability logs.raw will differ (short retention, many partitions, no keys). Same product, five contracts.

Related: foundations — scale, observability architecture, fraud architecture.

How to study this module¶

1. Read the log until "offset" and "segment" are muscle memory.
2. Read partitions with the simulator open. Predict lag before you generate it in the labs.
3. Read replication and be able to say what acks=all does when ISR shrinks to one.
4. Read exactly-once sceptically. Name the pipeline it actually covers.
5. Skim gotchas, then do the labs including now break it.
6. Continue into Flink with the same service-events topic.

You are done with Kafka as a platform when you can look at a lag graph, an ISR shrink, and a rebalance, and say which one is the incident.