Skip to content

Production Incidents

02:47 AM. P1 consumer_lag_seconds{topic="product-events"} > 180. Three teams' dashboards go stale in the next ten minutes and you have not opened a terminal yet.

A. Restart the consumer group. B. Add more processing slots. C. Check per-partition lag before touching anything else. D. Page the on-call lead and wait for guidance.

Only one of those survives contact with the metrics below — figuring out which one, before you act, is the whole exercise on this page. These are drills, not blog posts. Read Alert and Symptoms. Write a hypothesis. Only then open Resolution.

The skill is not memorising the root cause. It is forming a falsifiable story from metrics: which layer, which key, which hop. That is what on-call actually is.

Related: Kafka gotchas, Spark gotchas, how to study, labs.

How to use these

  1. Read the alert and the telemetry. Do not skim to the details tag.
  2. Pause. Name: system, failure class (skew, time, planning, memory, disk), what you would query next.
  3. Expand resolution. Compare. If you were wrong, write why the symptoms also fit your story — that is how you get better, not by flipping the card.

Every incident below is a shape you will meet in the five architectures. Kafka hot partition is analytics whales and fraud keys. Spark skew is e-commerce joins. Flink watermarks are IoT idle devices. ClickHouse parts/ORDER BY is every dashboard. Iceberg snapshots are cold paths. Trino coordinator OOM is "just one join."

Incidents 1–6 below are each contained inside one system's page, because that is how you learn the system. Production incidents rarely respect that boundary — the alert fires on a business metric, and the broken layer is three or four hops away from wherever the dashboard lives. The four cross-system incidents after them are the harder, more realistic drill: no page tells you which system to open first. The last of the four is a different category again — every system is correct, and the failure is in the interpretation.

Incident 1 — Kafka: lag on one partition

Alert

P1 consumer_lag_seconds{topic="product-events", group="flink-enrich"} > 180 for partition 7 only. SLO: dashboards < 60 s fresh.

Symptoms

  • kafka.consumer.lag partition 7: 4.2 million messages and climbing; partitions 0–6 and 8–23: < 200.
  • Flink subtask 7 CPU 95%, checkpoint duration 2×; other subtasks idle-ish.
  • Broker disk and network: leader of p7 is hot; other brokers fine.
  • Produce rate global: unchanged at 80k/s. No ISR shrink. No rebalance storm (rebalance_rate quiet).
  • kafka-consumer-groups.sh --describe: one consumer assigned p7+others in the group; lag not explained by fewer consumers than partitions (24 partitions, 24 subtasks).
  • Sample keys on p7: one customer_id=cust_0042 dominates (~70% of p7). Overall traffic: that tenant is ~12% of the company — enough to fill one hash bucket.
  • Other consumer groups on the same topic: same partition lagging.

Form a hypothesis

Is this (a) slow sink, (b) GC on one JVM, © key skew, (d) broker disk, (e) consumer count? What would disprove key skew in 30 seconds?

Resolution — Kafka hot partition ### Root cause Partition key is `customer_id`. Tenant `cust_0042` (or a bot, or `service=api-gateway` in observability) hashes to partition 7. Kafka **cannot** split a partition across consumers in a group. Extra Flink slots will not help p7. This is [hot partition](../kafka/gotchas.md) + [analytics noisy neighbour](../architectures/analytics-platform.md). Not a global produce outage (other partitions healthy). Not a ClickHouse sink (that would lag **all** partitions unless the sink is keyed the same way — and even then, check keys first). ### Fix (emergency) 1. **Quota / sample** that tenant at the gateway if product allows — stops the bleeding. 2. Scale **processing of that key**: a **separate consumer group** with a dedicated job that only reads a **new** topic for the whale, or a Flink sidecar — still one partition until you re-key. 3. Short-term re-key is a **new topic** (or new key) because changing keys mid-topic does not move old data. Sustainable keying: - Compound: `hash(customer_id) % N` **only for known whales**, suffix `cust_0042#0..15` so 16 partitions share the load; downstream re-aggregates. - Do **not** round-robin **all** keys — you break per-tenant ordering and Flink keyed state locality for everyone. ### Prevention - Alert **lag max / lag p50 ratio** per group, not only sum lag. - Dashboards: produce bytes **per partition**. - Tenant quotas. - Capacity: if one customer is 40% of traffic, they are an architecture, not a row. See [Kafka partition sim](../simulations/kafka-partitions.html). - Lab: [Kafka lab](../labs/index.md) hot-key exercise.

Incident 2 — Spark: executor OOM on a skewed join

Alert

Spark job fct_orders_enrich failed. ExecutorLostFailure: Exit code 137 (OOMKilled). Airflow retry also failed. SLA: 07:00 gold table.

Symptoms

  • Spark UI: stage SortMergeJoin — 199 tasks succeed in 40 s, one task runs 25+ min then dies. Task is partition 118.
  • Shuffle read on that task: 180 GB; others ~1–2 GB. Peak execution memory at cap. Spill to disk huge then OOM.
  • Driver fine; not a collect().
  • AQE enabled; spark.sql.adaptive.skewJoin.enabled true — still died (skew larger than AQE split budget / join type).
  • Input: events 2 TB, customers 8 GB. Join key customer_id.
  • events histogram (from a sample job): customer_id=cust_wholesale ~18% of rows (bot + marketplace).
  • Config: spark.sql.shuffle.partitions=200. Executor 16 GB, 4 cores.
  • GC logs: old gen full on that executor only.

Form a hypothesis

Driver OOM vs executor skew vs exploding join vs too few partitions globally? What number in the UI splits those?

Resolution — Spark skewed join ### Root cause Sort-merge join shuffled on `customer_id`. One key landed in one reducer. That reducer built a huge join relation and died. Classic [data skew](../spark/gotchas.md). AQE skew split **can** fix moderate skew; 180 GB vs 2 GB is a **salt or broadcast** problem. Not exploding join (that would blow **output** row counts on many tasks if many keys duplicate). Not "need 4000 executors" (they would sit idle). ### Fix 1. If `customers` is unique on `customer_id` and **small**: `broadcast(customers)` — **eliminates** the big shuffle. 8 GB broadcast is OK with memory; 80 GB is not. 2. If the whale is few keys: **salt** the join key on the large side, explode salt on the small side, drop salt after. 3. Two-path join: pull whale keys to a separate job; join the rest normally. 4. Raise executor memory **only** as a bridge — it does not fix 10×. Emergency to hit 07:00: filter out the whale key, publish with a quality flag, backfill the whale. ### Prevention - Pre-job: `groupBy(key).count()` approx on join keys; alert on max/median. - Broadcast threshold honest; stats up to date. - See [shuffle sim](../simulations/spark-shuffle.html); [Spark lab](../labs/index.md) skew exercise. - Product: bot `customer_id` should not share the key space with paying tenants.

Alert

flink.job.records_out for iot-1min-rollups is 0 for 40 minutes. Input records_in still 300k/s. Downstream Grafana 1-minute IoT panels frozen (last point old). Checkpoints succeed.

Symptoms

  • Flink UI: watermark for most keys not advancing; one Kafka partition shows source idle time 0 but the event-time watermark is stuck at T-6 hours.
  • A subset of devices send clocks in the past (bad firmware) or a dead producer still holds a partition with no records — depending on strategy, watermark is min over partitions.
  • Tumbling event-time 1-minute windows never close.
  • Processing-time operator counters still move (you added a debug map).
  • Late-event side output empty (windows never close, so "late" is undefined).
  • Kafka lag low — this is not incident 1.
  • Checkpoint size stable (state not exploding).

Form a hypothesis

Slow sink vs watermark vs key skew vs checkpoint? Which UI number is the watermark?

Resolution — Flink watermark stall ### Root cause Event-time windows **close on watermarks**, not on wall clock. Watermark was `min` over Kafka partitions (or over keys without idleness). One partition silent (device gateway down) **or** one partition of very old timestamps **held the min down**. All windows waited. Checkpoints succeeding proves the job is "healthy" in a useless sense. This is [IoT](../architectures/iot.md) and [Flink time](../flink/time.md). ### Fix 1. Enable **source idleness** (`withIdleness(Duration.ofMinutes(1))`) so silent partitions do not stall the world. 2. Filter/clamp **impossible timestamps** (e.g. > 1 h skew) to a side output; do not let year-2019 firmware block 2026 windows. 3. For "latest value" product, consider **processing-time** or a hybrid flush. 4. Temporarily restart with a watermark strategy that ignores idle partitions; expect a burst of window output. ### Prevention - Alert: `current_watermark` vs `now()` (allowed skew SLO). - Alert: `records_out == 0 && records_in > 0` for windowed jobs. - Lab: [Flink lab](../labs/index.md) idle/event-time exercise. - Do not use event-time windows on a source you cannot watermark.

Incident 4 — ClickHouse: query 10× slower

Alert

Grafana P2 dashboard_query_seconds{panel="error_rate"} p95 12 s (was 0.8 s). On-call cannot page by service. Inserts still succeeding.

Shape A — wrong ORDER BY after a "cleanup" migration

  • EXPLAIN / system.query_log: marks read ≈ all marks; selected_parts large.
  • Table events now ORDER BY (timestamp, service) (someone "optimised for time").
  • Query: WHERE service = 'api-gateway' AND timestamp > now() - 15 min.
  • primary_key cannot skip by service; must scan the 15-minute range across all services. At 5M events/s that range is huge.
  • Compression and CPU up; disk IO up.

Shape B — too many parts

  • system.parts for events: tens of thousands of active parts.
  • Insert via Kafka engine row-ish or tiny batches every 50 ms.
  • system.metrics BackgroundMerges saturated; DelayedInserts > 0 occasionally.
  • Queries open too many files (Too many parts warnings in log).
  • ORDER BY still (service, timestamp) — EXPLAIN would skip marks if parts were few.

Form a hypothesis

Would you look at query_log marks first or system.parts? How do the two shapes differ in Grafana (all queries vs some)?

Resolution — ClickHouse ORDER BY or parts ### Root cause **A:** Sparse index follows `ORDER BY`. Prefix of the key must match filters. Service-first is the [observability](../architectures/observability.md) and [SaaS](../architectures/analytics-platform.md) dashboard. Time-first is the global SRE scan. Migration inverted the product. See [ORDER BY sim](../simulations/clickhouse-order-by.html). **B:** Each tiny insert creates a part. Merges cannot keep up. Queries concatenate thousands of index pieces. Same symptom (slow SELECT), different table. ### Fix **A:** New table with `ORDER BY (service, timestamp)` (or `(customer_id, timestamp)`), insert-select or dual-write, swap. Cannot "just ALTER" the physical key cheaply on huge data. **B:** Batch inserts (1–10 s). Pause Kafka engine, `OPTIMIZE TABLE ... FINAL` only as emergency (it is expensive). Raise insert block size. Fix the producer. ### Prevention - `query_log` dashboard: marks read vs total. - Alert `parts` count per table. - Lab: [ClickHouse lab](../labs/index.md). - Change `ORDER BY` is a **design review**, not a tidy-up.

Incident 5 — Iceberg: snapshot accumulation, slow planning

Alert

Spark/Trino jobs on lake.events planning 15–40 minutes (was 2). Queries that read 1 day still plan like they must inspect the world. S3 LIST/GET on metadata spiking. Compute idle during plan.

Symptoms

  • snapshots table: 180k snapshots, expire never run. Hourly job every 3 minutes (mis-cron) committed for months.
  • Manifest lists huge; many small data files (5–20 MB) in recent partitions.
  • rewrite_data_files not scheduled; expire_snapshots commented out "until we need time travel."
  • Trino: coordinator CPU in planning, workers idle, then a burst.
  • Time travel "last 2 hours" still works — too well: you kept everything.
  • Disk/S3 metadata: billions of objects in the prefix.

Form a hypothesis

Slow scan of data vs slow planning? Which metric (worker bytes vs coordinator CPU / S3 GET on metadata/)?

Resolution — Iceberg snapshot pile ### Root cause Every commit adds a snapshot. Iceberg planning walks metadata (manifests) to pick files. Unexpired snapshots + tiny files + missing compaction → **metadata amplification**. The lake is correct; it is unusable. Cold path of [observability](../architectures/observability.md) / [analytics](../architectures/analytics-platform.md). This is not "Trino is slow at SQL" and not "need 200 workers" (they are idle in plan). ### Fix 1. **Expire snapshots** older than the time-travel policy (e.g. 7 d), retain last N. 2. **Rewrite manifests** / `rewrite_data_files` on hot partitions. 3. Stop the 3-minute commit storm; batch commits. 4. Orphan file removal **after** expire, carefully. Emergency: query a **specific snapshot** you know is healthy; do not `SELECT` through current if metadata is pathological — still may plan slowly. Worst case: read known file list from a **new** table created from a recent snapshot's data files (surgical). ### Prevention - Compaction + expire as **Airflow gold**, not a wiki. - Alert snapshot count, files per partition, planning time. - Time travel is a **retention policy**, not infinity ([security](../security/index.md) GDPR). - Fewer, larger commits.

Incident 6 — Trino: coordinator OOM on a wild join

Alert

Trino cluster down. Coordinator java.lang.OutOfMemoryError. All users fail. Grafana on CH unaffected.

Symptoms

  • Last query in log:
    SELECT * FROM iceberg.events e JOIN postgres.prod.users u ON e.user_id = u.id
    with no time filter, events is 2 years, users 40 million rows.
  • Query: distributed join, broadcast hinted or estimated wrong (stats missing on Iceberg). Coordinator or a worker tried to hold a huge build side; coordinator OOM often from query plan / metadata / buffered results or a join that was scheduled as single-node.
  • JOIN on cast(user_id AS varchar) disabling pruning.
  • Cost-based optimizer thought events was 0 rows (stale stats).
  • Exchange REPARTITION vs BROADCAST in EXPLAIN: broadcast of a large side.
  • Other queries died because one coordinator.

Form a hypothesis

Worker scan vs coordinator? What EXPLAIN line is the smoking gun?

Resolution — Trino coordinator OOM ### Root cause Unbounded lake ⋈ OLTP replica. Missing partition predicate. Bad stats → broadcast or a too-fat plan. Coordinator is a **SPOF** for planning and sometimes for results. This is why [CH vs Trino](../comparisons/clickhouse-vs-trino.md) says Trino is not the 100 ms UI and not a playground without governors. ### Fix 1. Restart coordinator (cluster is already dead). 2. Kill the query pattern: **require** `WHERE e.date > ...` via view or policy. 3. Disable broadcast for huge tables; set `join_distribution_type`. 4. Memory limits per query; **isolation** (separate Trino for ad-hoc vs gold). 5. Do not `SELECT *` the lake. ### Prevention - Query max memory, timeout, **max scanned bytes**. - Stats collection on Iceberg. - Analysts default to **sampled** views. - Two coordinators/clusters: exploratory vs production ETL. - Education: join **day** of events to users, not history. - [Security](../security/index.md): this query may also be an exfil; audit it.

Cross-system incident 1 — Revenue dashboard down 22%, everything green

Alert

09:10. Executive revenue dashboard (ClickHouse, fed from events_agg) shows -22% vs the same hour last week. No page fired — this was noticed by a human, not an alert, which is itself the first clue.

Symptoms

  • ClickHouse events_agg freshness: normal (max(ts) >= now() - 15m).
  • Flink checkpoint duration and backpressure: normal.
  • Kafka consumer lag on order-events: normal, near zero on every partition.
  • Iceberg raw_events row count for the last 24h: normal, matches the usual day-over-day pattern.
  • CDC source reconciliation (Debezium row count vs source Postgres row count for orders): -3% — the one number that is not normal, and the smallest deviation of the bunch.

Form a hypothesis

Every downstream layer says "normal." Where do you look next, and what hypothesis explains a small (-3%) discrepancy at the source feeding a large (-22%) discrepancy at the dashboard? What would disprove it in the next five minutes? A specific prompt: the CDC connector restarted cleanly last night after database maintenance and has reported healthy ever since — what could a clean restart still have lost?

Resolution — revenue drop, cross-system ### Root cause The -3% CDC reconciliation gap is not "3% of orders missing at random" — it is a contiguous window of changes missing entirely, and it happened because **CDC recovery state was recreated instead of preserved**. During PostgreSQL maintenance the previous night, the logical replication slot for the Debezium connector was accidentally dropped (it was blocking a WAL cleanup and someone removed it to let the maintenance proceed). The connector restarted afterward and a **new** slot was created at the current WAL position, while Kafka Connect's stored offset for the connector still referenced an older LSN. Changes between that older LSN and the newly created slot's start position exist in neither place: the old slot that was retaining them is gone, and the new slot only begins streaming from where it was created. Debezium resumed cleanly, reported healthy, and streamed everything from that moment forward. This is the important correction to make to most people's mental model: Debezium's *normal* restart behaviour is safe. It persists LSN offsets and resumes from the previously recorded position, and a brief lag during maintenance is exactly the case it is designed to survive. The dangerous case is not lag — it is when the **replication slot itself** is recreated, upgraded around, or otherwise loses the retention that the stored offset depends on. Debezium's own documentation warns that a recreated slot can make older changes unavailable and lead to skipped events. 
Kafka Connect offset  ──┐
                        ├── these two must refer to the SAME retained WAL history
Postgres replication ───┘
        slot

Drop the slot, keep the offset  →  a silent, unrecoverable gap between them.
Every layer *downstream* of Kafka reports "normal" because each of them is measuring throughput and freshness of what **did** arrive — none of them can see what never left the source. The missing window happens to be concentrated in the company's highest-value enterprise segment (a handful of large customers whose orders dominate revenue $ even though they're a small fraction of order *count*), which is why a 3% row gap becomes a 22% revenue gap. The chain a learner must walk is: **business metric (revenue) → serving layer (ClickHouse, fine) → transformation (Flink, fine) → stream processor (Kafka, fine, lag=0) → transport (CDC/Debezium) → source (Postgres)**. Every "fine" reading upstream of the actual break is fine *because it correctly reflects what it received* — the break is the one hop nothing downstream can observe at all: data that never entered the pipeline. ### Fix 1. Establish the gap's boundaries: the connector's stored Kafka Connect offset LSN (the last change it is *sure* it consumed) versus the new slot's `confirmed_flush_lsn` at creation. Everything between them is the missing window; map it to a wall-clock range using the maintenance timeline. 2. Backfill that window with a bounded, filtered snapshot re-read of `orders` over the affected time range (see [CDC — reconciliation](../foundations/cdc.md#reconciliation)) — an incremental/ad-hoc snapshot, not a full re-snapshot of the table. 3. Reprocess only the affected hours through Flink → Iceberg → ClickHouse; do not reprocess the whole day. 4. Do **not** "fix" it by dropping and recreating the slot again — that is the action that caused the incident. ### Prevention - **Never casually recreate CDC state.** The stored connector offset and the replication slot are *one recovery contract*, not two independent resources. Deleting either one alone destroys the guarantee that the pair provides. Treat "drop the replication slot" the way you treat "drop the table": a change-controlled action with an explicit plan for the data it strands. - Put replication slots in the runbook for every database maintenance and upgrade, with an owner. The most common route to this incident is a DBA clearing a slot that is "blocking WAL cleanup" without knowing a pipeline depends on it — so also alert on slot *WAL retention* long before it becomes an emergency someone resolves by deleting it. - Alert on the *existence and identity* of the slot, not just consumer health: a connector reporting healthy against a freshly created slot looks identical to one resuming correctly. - Reconciliation between source row count and CDC-consumed row count, **alerting**, not just dashboarded — the -3% here was visible before the incident but nobody was paged on it. - A revenue metric broken down by segment in the alert itself, so "-22% overall, -0% for 95% of segments" surfaces the concentration immediately instead of requiring a human to notice and dig. - The general lesson: "every layer is green" proves every layer is internally consistent with what it received — it proves nothing about what never arrived. See [metadata — declared vs observed truth](../metadata/index.md#contracts-vs-catalogues-declared-truth-vs-observed-truth).

Cross-system incident 2 — Pipeline green, data wrong

Alert

Airflow: SUCCESS. Spark job: SUCCESS. Iceberg snapshot committed. ClickHouse ingestion: complete, freshness normal. Conversion-rate dashboard: +400% overnight, with no marketing change and no traffic spike.

Symptoms

  • Every system-level health check is green — this is the point of the drill. A pipeline that fails loudly is not the hard case.
  • Row counts at each stage (Iceberg raw → Iceberg clean → ClickHouse agg) are all higher than usual by roughly the same factor the metric is inflated by.
  • The conversion-rate SQL is count(purchase_events) / count(session_events), joined on session_id.
  • A schema change shipped two days ago added a retry_count field to the purchase-event producer for client-side retry visibility.

Form a hypothesis

List every plausible cause consistent with "every system reports success, but the specific numerator/denominator of one ratio is wrong": duplicate replay, join explosion, late-arriving data, dimension duplication, a semantic schema change, an incorrect denominator. Which ones does the row-count-inflation clue eliminate, and which metric would you compute next to eliminate the rest?

Resolution — pipeline green, data wrong ### Root cause The client-side retry logic added with `retry_count` was implemented as "resend the same purchase event with the same `session_id` on any client-side timeout," including timeouts where the original request actually succeeded server-side. Purchase events are not deduplicated by event id anywhere in the pipeline — Kafka producer idempotence prevents *broker-level* duplication of a single publish call, but does nothing about the application making a second, distinct publish call with the same business meaning. Every stage's row count inflated by the same factor because every stage faithfully processed every row it received — the duplication happened **before** stage 1 of [the correctness chain](../reference/correctness-invariants.md), at Produced→Accepted, and nothing downstream had an invariant that would have caught a business-level duplicate with a fresh producer sequence number. The elimination sequence: duplicate replay (Kafka-level) is ruled out because consumer offsets are clean and Flink/Spark checkpoint state shows no replay events; join explosion is ruled out because the `session_events` denominator inflated by the *same* factor, not independently; late-arriving data doesn't explain a sustained, uniform inflation; dimension duplication would show as a join fan-out on one specific dimension key, not a uniform multiplier; incorrect denominator alone wouldn't move both numerator and denominator together. What's left, and what the row-count-inflation pattern actually points to, is duplicate *rows entering upstream of any dedup logic* — a producer-side change. ### Fix 1. Add an idempotency key (a client-generated request id, not `session_id`) to the purchase-event schema and dedupe on it at the earliest possible stage — ideally the producer, backstopped by a dedup step in Flink. 2. Backfill the affected window by deduplicating the existing Iceberg data on the retry-safe key, once the producer team confirms which field actually identifies a unique attempt. 3. Add a schema-change gate: a client-side retry change is a semantic change to event identity, and should have gone through the same [data contract](../foundations/data-contracts.md) review as a schema-breaking change, even though the schema *technically* only added a field. ### Prevention - A reconciliation check comparing purchase-event count against an independent source (payment processor's own count) — the "what would you measure" question this incident is really testing. - Treat "adds a field for observability" as a semantic-review trigger, not just a backward-compatible schema no-op — the field itself (`retry_count`) was evidence a producer behavior change had happened, and nobody read it that way.

Cross-system incident 3 — Dashboard 15 minutes stale, Kafka lag near zero

Alert

Dashboard freshness SLO breach: gold.events_agg is 15 minutes behind wall clock. Kafka consumer lag on the upstream topic: near zero across every partition.

Symptoms

  • Kafka lag: healthy. This alone rules out the most reflexive hypothesis ("Kafka is behind").
  • Flink job is consuming near-real-time; its own internal watermark is close to processing time.
  • The sink connector from Flink to Iceberg batches writes and commits a new snapshot only every 5 minutes.
  • ClickHouse ingests from Iceberg via a scheduled Airflow-triggered job, not continuously.
  • Airflow's ingestion DAG runs on a 10-minute schedule interval, not event-driven.

Form a hypothesis

Kafka lag near zero means events are being consumed promptly. It says nothing about how long each downstream hop takes to make an event visible in the final dashboard. Name every hop between "consumed from Kafka" and "visible on the dashboard," and estimate how many minutes each one could plausibly be contributing.

Resolution — latency without Kafka lag ### Root cause **Kafka lag measures time spent waiting to be consumed — it says nothing about time spent being processed, batched, committed, or scheduled after that.** Five minutes of Flink→Iceberg commit batching, plus up to 10 minutes of Airflow schedule interval before the next ClickHouse ingestion run picks up the new Iceberg snapshot, sums to the observed ~15-minute staleness — with Kafka lag at zero the entire time, because Kafka's job (deliver the event to a consumer promptly) was done correctly. This is the general lesson: end-to-end freshness is the **sum of every hop's own latency contract**, not the latency of whichever hop happens to have the most visible metric. ### Fix 1. Reduce the Flink→Iceberg commit interval if the workload's small-file cost (see [cost engineering — Iceberg](../reference/cost-engineering.md)) tolerates more frequent, smaller commits. 2. Move ClickHouse ingestion from a 10-minute Airflow schedule to an [Asset-triggered](../airflow/index.md#sensors-pools-mapping-slas-assets) run fired by the new Iceberg snapshot, removing the schedule-interval tax entirely. 3. Add a synthetic canary event at the true source, timestamped, and measure its arrival time at the dashboard directly — an end-to-end freshness metric, not a per-hop proxy. ### Prevention - Alert on **end-to-end freshness** (`gold.events_agg` freshness SLO, as in [metadata — freshness](../metadata/index.md#freshness)) as the primary page, with per-hop metrics (Kafka lag, Flink watermark, Iceberg commit age, Airflow run recency) as the drill-down, not the other way around. Paging on Kafka lag alone, as this incident shows, pages on the wrong layer entirely when the bottleneck moves downstream. - Name, for every pipeline, the latency budget each hop is allowed to consume, so "which hop ate the 15 minutes" is a lookup, not an investigation.

Cross-system incident 4 — Fraud rate up 40%, every system correct

This one is a different category from the three above. Incidents 1–3 were deterministic pipeline failures: something broke, and the job was to find it. Here nothing is broken, and that is the entire lesson.

Alert

11:40. The fraud team's daily monitor: fraud_rate up 40% week-over-week, from 0.5% of transactions to 0.7%. The fraud model's alerting thresholds were tuned at 0.5%, so the review queue has tripled and two analysts are drowning. Engineering is paged to find the pipeline bug.

Symptoms

Everything checks out — and the on-call engineer checks thoroughly, which is why this incident takes four hours:

  • Kafka: lag zero, no partition skew, no producer errors, no gaps in offsets.
  • Flink: checkpoints healthy, watermark tracking processing time, no late-data drops, no state restore since last deploy.
  • Iceberg: row counts reconcile against Kafka offsets exactly. No duplicate transaction IDs. No missing hours.
  • ClickHouse: events_agg matches an independent Trino query over the same Iceberg snapshot, to the row.
  • CDC reconciliation against source Postgres: 0% gap.
  • No deploys to the fraud model, the feature pipeline, or any transformation in the last 14 days.
  • Replaying last week's Kafka data through today's pipeline reproduces last week's 0.5% exactly. Replaying this week's data reproduces 0.7%.

Form a hypothesis

Every correctness check passes, including a replay that proves the pipeline is deterministic and unchanged. The data is correct. Before reading on: if the pipeline is right and the number moved anyway, what kinds of things can still have changed? Name at least two, and name the query you would run to distinguish them.

Resolution — correct data, wrong conclusion ### Root cause **The pipeline is fine. The population changed.** Break `fraud_rate` down by customer segment and the aggregate 40% rise disappears into something much less alarming: 
                        last week          this week
segment            txns   fraud_rate    txns   fraud_rate
─────────────────────────────────────────────────────────
SMB self-serve     980k       0.50%      985k       0.50%   ← unchanged
Enterprise (old)    20k       0.40%       20k       0.40%   ← unchanged
Enterprise (NEW)     —           —       120k       1.40%   ← new customer, onboarded Monday
─────────────────────────────────────────────────────────
TOTAL             1000k       0.50%     1125k       0.71%
Not one segment's fraud rate moved. A single new enterprise customer went live on Monday — a marketplace with a fundamentally different traffic profile (guest checkout, high-value electronics, international cards) whose *inherent* fraud rate is legitimately ~3× the platform average. It now contributes 11% of transaction volume. The aggregate moved because the **mix** moved. This is [Simpson's paradox](https://en.wikipedia.org/wiki/Simpson%27s_paradox) territory, and it is one of the most common ways a perfectly healthy data platform produces a false alarm — or, far worse, hides a real one. The same mechanism runs in reverse: a growing low-fraud segment can *mask* a genuine fraud spike in an existing segment, and every system will still be green. The conclusion, not the data, was wrong: 
correct data  ≠  correct interpretation
### Fix There is no pipeline fix, and saying so clearly is the deliverable: 1. Report the finding as a **mix shift**, with the segment breakdown above, not as "no bug found." "Everything is green" closes a ticket; the breakdown changes a decision. 2. Hand it to the fraud team as their decision: the model's 0.5% threshold was calibrated on a population that no longer exists. Either segment the threshold, or accept the new baseline — a modelling and risk call, not an engineering one. 3. Add the segment dimension to the fraud monitor so the *next* version of this question answers itself. ### Prevention - **Monitor rates alongside their denominators.** A ratio alert with no volume or mix context will fire on population change as readily as on the thing it was built to detect. Show `fraud_rate`, transaction count, and segment mix on the same panel. - Alert on **within-segment** rates, and treat the aggregate as a summary, not a signal. - When traffic composition changes materially — a large customer onboards, a region launches, a pricing tier ships — flag it as a known event on analytics dashboards, the same way deploys are annotated. Business changes are as load-bearing for a metric's interpretation as code changes. - Add "what changed in the *population*?" to the incident checklist next to "what changed in the *code*?" Four hours went into proving the pipeline correct because nobody asked the second question first. ### Why this incident is in the curriculum It marks a boundary that data engineers are repeatedly asked to stand on: 
data engineering     did the right bytes arrive, intact, on time?     ✓ yes
analytics            is the metric computed as defined?               ✓ yes
statistics           is the comparison valid across two populations?  ✗ NO
business semantics   does the number mean what the reader assumes?    ✗ NO
Your job ends at the first two lines — but the *page* you receive will always be phrased as if the failure is in your half. An engineer who can only answer "the pipeline is correct" leaves the organization exactly where it started. An engineer who can produce the segment breakdown, name the mix shift, and hand it to the right owner has actually resolved the incident. The skill is knowing which of the four rows above broke, and being able to demonstrate it.

Cross-walk

Incident Architecture Lab / sim
Kafka one-partition lag Analytics whale, fraud key, observability hot service Kafka lab, partition sim
Spark join OOM E-commerce enrich, fraud labels Spark lab, shuffle sim
Flink no output IoT rollups, session windows Flink lab
CH 10× Every Grafana CH lab, ORDER BY sim
Iceberg planning Cold paths Lakehouse module
Trino OOM Ad-hoc federation CH vs Trino
Revenue drop, all-green SaaS analytics, CDC reconciliation CDC, correctness invariants
Pipeline green, data wrong Any pipeline with client-side retries correctness invariants, data contracts
Stale dashboard, zero Kafka lag Every multi-hop pipeline Flink checkpoints, Airflow
Fraud rate up 40%, all systems correct Fraud detection, SaaS analytics metric definitions, data modelling

After-action habit

For your own incidents, steal this structure: Alert → Symptoms (numbers) → Hypothesis pause → Root cause → Fix → Prevention. If prevention is "be careful," you have not finished. Add a metric and an owner.