Interactive Simulations¶

Browser tools for failure modes that are hard to feel from prose: hot Kafka keys, shuffle skew, ClickHouse ORDER BY, TSDB cardinality, backpressure recovery time, stalled watermarks, ISR failure, metadata pruning, hash distribution, and dimension history. They are not dashboards of a real cluster. They are labs for your mental model.

Related: how to study, labs, incidents.

For a single ordered route through all four kinds of practice, use the practice map. Each concept page also links directly to its matching simulation before the final understanding check.

A useful way to explore¶

1. Read the concept page first (linked below). If you open the sim cold, you will slide sliders.
2. Predict what the default settings show. One sentence, out loud or on paper.
3. Change one variable (hot-key %, skew, ORDER BY, a label).
4. Compare to the prediction. Name the metric that would page you in production.
5. Only then try the "what to try" recipe on this page.

The five difficulty levels¶

There will not be more simulators — thirteen is enough surface area. What there is room for is going deeper into each one. Every sim on this page supports the same five-level ladder, and level 5 is the one that actually changes how you design systems:

Level 5 is the point of the whole page. Levels 1–4 can all be satisfied by turning a slider up, which is exactly the habit production punishes: it is the fix that costs money, arrives slowly, and often just relocates the bottleneck. Ban the hardware lever and the only remaining moves are the ones that change the shape of the problem — re-key the data, change the ORDER BY, shard the hot key, batch the writes, drop a dimension, move work to a different hop, or decide the requirement itself was wrong.

If a level-5 answer does not exist for a sim's failure mode, that is a finding too: some constraints genuinely are physical, and knowing which ones is why you would spend the money.

These are single HTML files. They work offline. In MkDocs they are linked pages. You can also open them from docs/simulations/ in the repo.

Kafka partition simulator¶

Open: kafka-partitions.html

Teaches: Kafka parallelism is partitions, not consumer count. Ordering and state locality are per key. A hot key is a hot partition. Extra consumers idle while one partition's lag explodes.

Four scenario-preset buttons (uniform, whale tenant, under-provisioned consumers, universally-too-slow consumers) load a named difficulty level with one click — use them to check a prediction fast, then go back to the sliders to push past them.

Read first: Kafka partitions, gotchas.

Predict before touching sliders:

• If consumers > partitions, how many consumers sit idle?
• If 70% of keys are one customer, can 12 consumers save you?

What to try:

1. Equal keys, 6 partitions, 6 consumers — lag ~flat.
2. 6 partitions, 12 consumers — idle consumers.
3. Hot key ratio 70% — one partition's lag grows. This is incident 1.
4. Increase partitions without changing the key — the hot key still hashes to one partition.

Production metric: lag per partition, produce bytes per partition, consumer CPU per subtask.

Architecture: SaaS whale, fraud user key.

Then run: Kafka lab with a real broker.

Spark shuffle visualiser¶

Open: spark-shuffle.html

Teaches: groupBy/join move data. Uniform keys → even reduce tasks. Skew → one reducer owns the stage. Salting splits a hot key on purpose.

Read first: shuffle.

Predict:

• Very skewed: what fraction of rows land on the busiest reducer?
• Does adding Spark executors split one key? (No.)

What to try:

1. Uniform — bar chart even; shuffle still happens.
2. Very skewed — one bar dominates; this is incident 2 without the OOM yet.
3. Salted — more bars, extra CPU, no 90% bar.

Production metric: Spark UI task duration / shuffle read max vs median.

Then run: Spark lab and watch stage 4040.

ClickHouse ORDER BY explorer¶

Open: clickhouse-order-by.html

Teaches: the sparse primary index is the physical ORDER BY. If WHERE does not match the prefix, you scan granules you did not think you would. Same SQL, 100% vs 0.3% marks.

Read first: ClickHouse.

Predict:

• Table ORDER BY (customer_id, timestamp), query one tenant last 15 min — skip or scan?
• Flip to ORDER BY (timestamp, customer_id) — same query?

What to try:

1. Scenario "wrong ORDER BY" vs "right" — compare granules read.
2. Imagine the observability query service + 15 min on a time-first table — that is incident 4 shape A.

Production metric: system.query_log marks / rows read, not "CPU is high."

Then run: ClickHouse lab with two real tables.

Cardinality calculator¶

Open: cardinality-calculator.html

Teaches: TSDB series count is the product of label values. Adding user_id (100k) turns a healthy metric into a memory bomb. Time on the x-axis does not justify Prometheus.

Read first: cardinality, TSDB vs OLAP.

Predict:

• 4 bounded labels (service, code, region, pod) — order of magnitude series?
• Same + user_id 1e5 — do you still have a TSDB?

What to try:

1. Start with 4 infra labels — stay under a comfortable series budget.
2. Add user_id or device_id — cross the line. That data belongs in ClickHouse.
3. Map to IoT 10M devices — Prometheus is for the cluster, not the fleet.

Production metric: Prometheus prometheus_tsdb_head_series, scrape duration, CH uniq(user_id) as a column cost not a series cost.

Backpressure & queue accumulation calculator¶

Open: backpressure-calculator.html

Teaches: a throughput mismatch does not vanish — it becomes a growing backlog. This is queue accumulation (dQ/dt = λ_in − λ_out), not Little's Law — the calculator turns "how long until we catch up" from a guess into arithmetic, and shows why a service rate that barely exceeds arrival still means a very slow recovery.

Read first: Backpressure & Queueing.

Four scenario-preset buttons (minor mismatch, major mismatch with strong scaling, "fixed" but barely above arrival, scaling that never exceeds arrival) let you jump straight to a named difficulty level.

Predict before touching sliders:

• Arrival 100k/s, service 70k/s for 20 minutes — roughly how many events back up?
• After scaling to 150k/s, does the system recover in minutes or hours?
• What happens if the "scaled" rate is set to exactly the arrival rate?

What to try:

1. Default mismatch — note the backlog number, then guess recovery time before scrolling to the verdict.
2. Set the post-scaling rate equal to arrival rate — watch recovery time diverge instead of hitting zero.
3. Set post-scaling rate just barely above arrival (e.g. +5%) — same backlog, very different recovery time from +50%.

Production metric: Kafka records-lag, Flink backpressured_time per operator, or a sink's write-latency trend — see debugging for which one separates which hypothesis.

Then run: the exercise at the bottom of Backpressure & Queueing with the numbers from this sim.

Watermark simulator¶

Open: watermark-simulator.html

Teaches: a downstream event-time watermark is the minimum across every active source split. One idle split with no idleness policy freezes the whole downstream watermark — and therefore every window waiting on it — forever, even while every other split advances normally.

Read first: Flink windows & watermarks, Flink time semantics.

Four scenario-preset buttons (balanced splits, one split far behind, idle split with no idleness policy, idle split with the policy enabled) load a named difficulty level in one click.

Predict before touching sliders:

• Split A's watermark is way ahead, split B stops advancing. Does the downstream watermark advance, stall, or split the difference?
• Turning on an idleness timeout for B — does that change what happens to a late event that still arrives on B?

What to try:

1. Both splits advancing evenly — downstream watermark tracks the slower of the two, windows close on schedule.
2. Freeze split B, no idleness policy — downstream watermark stalls at B's last value; a window waiting to close never does.
3. Enable the idleness timeout past B's stall duration — downstream watermark starts advancing again, using only A.

Production metric: per-source-split watermark lag, Flink's currentInputWatermark metric, and records_out per window operator flatlining despite RUNNING status.

Then run: Flink lab stalled_watermark.py, then check_event_time.py to see the same mechanism asserted, not just visualized.

Kafka ISR failure simulator¶

Open: kafka-isr-simulator.html

Teaches: acks, min.insync.replicas, and current ISR membership jointly decide whether a write succeeds, blocks, or throws NotEnoughReplicas — and acks=all only ever means "wait for the current ISR," which can have shrunk to just the leader.

Read first: Kafka replication & durability.

Four scenario-preset buttons (healthy ISR, ISR shrinks from a lagging follower, broker fails with ISR intact, broker fails with ISR empty) set up a named failure level in one click.

Predict before touching sliders:

• ISR is down to just the leader, acks=all, min.insync.replicas=1. Does this write have the same durability guarantee as acks=1?
• Kill the leader with an empty ISR and unclean election disabled — does the partition lose data, or go unavailable?

What to try:

1. Full ISR, acks=all, min.insync.replicas=2 — healthy write, safe leader failover.
2. Shrink ISR to the leader only, same settings — writes start failing with NotEnoughReplicas rather than silently losing durability.
3. Kill the leader with unclean election enabled on an empty ISR — a replica with stale data becomes leader; flag the data-loss window explicitly.

Production metric: UnderReplicatedPartitions, ISR shrink/expand broker logs, NotEnoughReplicasException rate on producers.

Then run: Kafka lab's broker-stop exercise, this time watching ISR membership instead of just consumer lag.

Iceberg manifest explorer¶

Open: iceberg-manifest-explorer.html

Teaches: Iceberg answers "what is the table right now" by reading a small tree of metadata (snapshot → manifest list → manifests → per-file stats), never by listing the object store — and a query prunes files using only that tree, before opening a single data file.

Read first: Apache Iceberg, Object Storage Internals.

Predict before clicking:

• After "commit a new write," does the previous snapshot stop being queryable?
• A date = filter that matches only 1 of 8 data files — how many files does the engine actually have to open to know that?

What to try:

1. Commit a second write — watch a new snapshot appear while the old one (and its manifests) stay intact for time travel.
2. Run a query with a narrow date = filter — watch most files go grey (skipped via manifest stats) without ever being opened.
3. Compare "files opened" to "total files" as you add more snapshots — pruning cost stays a metadata read, not a data scan.

Production metric: Iceberg's plan.metadata-read duration and manifest count in query plans; a growing manifest count with no compaction is the thing that eventually makes even metadata-only planning slow.

Then run: Iceberg's own worked example against a real table to see metadata.json and manifest files on disk.

Parquet row-group explorer¶

Open: parquet-row-group-explorer.html

Teaches: predicate pushdown skips row groups using only the file footer's min/max stats — and that only works if the data was actually sorted on the filtered column before write. Partitioning and row-group statistics are two different pruning mechanisms; this simulator has no partitioning at all.

Read first: Parquet Internals.

Predict before touching the toggle:

• Data sorted by customer_id before write, filter on one customer_id — how many of 8 row groups get opened?
• Flip to unsorted (random write order), same filter — does the count change, and why?

What to try:

1. Sorted mode, narrow filter — most row groups skip; only 1-2 opened.
2. Unsorted mode, same filter — nearly all row groups' min/max ranges span the whole domain, so almost none skip.
3. Compare this to a partitioned-but-unsorted file: partitioning alone would have pruned by directory, but rows inside the partition are still unsorted, so row-group pruning still fails at the file level.

Production metric: explain() on Spark/Trino showing ReadSchema/PushedFilters and the row-group count actually scanned vs. total.

Then run: Spark's Catalyst page explain("formatted") example against a real Parquet table you control the write-order of.

Bloom filter playground¶

Open: bloom-filter-playground.html

Teaches: a bloom filter trades a small, tunable false-positive rate for a large space saving over storing the real set — and that rate is a live function of bit-array size, hash-function count, and how full the filter already is, not a fixed constant you can ignore.

Read first: Why Columnar Storage (skip indexes), ClickHouse.

Predict before adding items:

• At a mostly-empty bit array, can a membership test for something never added ever return "possibly present"?
• What happens to the false-positive rate as you keep adding items without growing the bit array?

What to try:

1. Add a handful of items to a large bit array — false-positive rate stays near zero.
2. Keep adding items without resizing — watch the live false-positive rate climb per the formula, not your intuition.
3. Use "find a false positive" once the filter is fairly full — confirm a real false positive, then note it is a wasted read, never wrong data.

Production metric: ClickHouse/Parquet bloom-index false-positive rate in practice vs. the bytes saved — a filter sized for yesterday's cardinality silently degrades as a table grows.

Consistent hashing visualizer¶

Open: consistent-hashing-visualizer.html

Teaches: consistent hashing on a ring moves roughly 1/N of keys when a node joins or leaves; naive hash(key) % N reshuffles nearly everything on the same event. Virtual nodes fix the load imbalance a small number of physical nodes has on a ring.

Read first: Partitioning (hash partitioning), Cassandra & ScyllaDB.

Predict before adding a node:

• Consistent hashing, add one node to a 5-node ring — roughly what fraction of keys change owner?
• Naive mod-N hashing, same change — same fraction, or nearly all keys?

What to try:

1. Add/remove a node under consistent hashing — note the small, bounded number of reassigned keys.
2. Toggle naive mod-N hashing, repeat — nearly every key reassigns.
3. Raise virtual nodes per physical node — watch per-node load bars even out from lopsided to balanced.

Production metric: Cassandra/DynamoDB partition-count rebalance events and per-node request-rate variance after a scale-out.

DynamoDB hot-key simulator¶

Open: dynamodb-hot-key-simulator.html

Teaches: a logical key hashes to exactly one physical partition, each partition gets a baseline slice of table throughput, and adaptive capacity lets a hot partition borrow throughput idle partitions are not using. Adaptive capacity reduces — but does not eliminate — a pathological hot key: a single key can still hit a per-partition ceiling that no amount of table capacity lifts.

Read first: DynamoDB.

Predict before touching sliders:

• Table utilization at 20% overall, one key gets 90% of traffic — throttled or fine, with adaptive capacity ON?
• Does raising total table throughput fix that one hot partition? Does it still help once the partition is at its ceiling?

What to try:

1. Adaptive capacity OFF, low overall utilization, moderate skew — the hot partition redlines at its fixed baseline slice while others sit idle. This is the old static model, and still the right worst case to reason about.
2. Adaptive capacity ON, same settings — the hot partition absorbs spare table capacity and the throttle clears. Watch the amber bars: those partitions are running above baseline and only surviving on borrowed throughput.
3. Push skew toward 100% with adaptive capacity still ON — the hot partition pins at the per-partition ceiling and throttles anyway. Raise total table throughput and confirm it changes nothing: spare capacity is irrelevant once one partition is at its ceiling.
4. Enable "shard the hot key across N suffixes" — the same logical entity's traffic spreads across N physical partitions, each with its own baseline and its own ceiling, and the throttle clears for a reason that survives at any scale.

Production metrics: you cannot read per-physical-partition consumed capacity directly — DynamoDB deliberately abstracts the partition away — so hot-key behaviour is inferred from several signals together: ThrottledRequests / WriteThrottleEvents, SuccessfulRequestLatency, table-level ConsumedCapacity (to prove the table is not saturated), CloudWatch Contributor Insights (most-accessed keys), and key-level application telemetry. The lesson is the inference, not the metric: when throttles rise while table capacity looks healthy, the missing variable is always distribution.

SCD Type-2 timeline explorer¶

Open: scd2-timeline-explorer.html

Teaches: an SCD Type-2 dimension keeps every historical version of a row with effective_from/effective_to/is_current, so "what was true as of date X" is answerable — but only because every change appends a new row instead of overwriting the old one.

Read first: Analytical Data Modelling.

Predict before recording a change:

• After changing plan from "free" to "pro," does the old row disappear, or stay with a closed effective_to?
• Querying "as of" a date before the change — which row answers that, the current one or a historical one?

What to try:

1. Record one change — watch a new row append while the prior row's effective_to closes and is_current flips.
2. Set "as of" to a date before the change — confirm the historical row, not the current one, is selected.
3. Contrast with what an SCD Type-1 (in-place update) table would show for the same "as of" query: today's value, applied retroactively to history — the silent-rewrite failure mode.

Production metric: a fact-to-dimension join using is_current = true instead of an effective_from/effective_to range join on the fact's event time — this is the bug this simulator is built to make visible.

What each sim is not¶

Level 5 for each sim: fix it without more hardware¶

Work these after you can already do levels 1–4 on the sim in question. Each one has at least one good answer that costs nothing but a design change.

Notice the pattern in the right-hand column: almost every entry is either change the key, change the layout, or change the requirement. That is a short list, and it is most of data architecture.

Suggested afternoon¶

1. Kafka sim → Kafka lab → incident 1.
2. Shuffle sim → Spark lab → incident 2.
3. ORDER BY sim → CH lab → incident 4.
4. Cardinality sim → rewrite one Prom metric at work that should be an event.
5. Backpressure sim → estimate a real recovery time from a lag incident you've seen at work.
6. Watermark sim → Flink lab stalled_watermark.py → check_event_time.py.
7. Kafka ISR sim → re-read the Kafka lab's broker-stop step with ISR in mind.
8. Iceberg + Parquet explorers → pick one real table at work and predict its pruning before running EXPLAIN.
9. Bloom filter + consistent hashing → connect to whichever KV/wide-column store you actually operate.
10. DynamoDB + SCD2 → the two "silent until it's an incident" failure modes: a throttled partition and a fact joined to the wrong dimension version.

If you only do the HTML, you will remember the pictures. If you do lab + incident, you will remember the metric. If you do the level-5 pass above, you will remember the design.

Contributing simulations¶

Keep them:

• One HTML file, no npm build.
• One concept, one failure mode.
• Sliders that map to real knobs (partitions, skew, labels).
• Dark-theme consistent with the site.

Do not add a sim that only animates a logo. If it cannot support a predict → compare loop, it is decoration.

Predict prompts (copy onto paper)¶

Kafka: "With 8 partitions and 8 consumers, lag is even. I now set hot key 80%. I predict partition i (whichever hashes the hot key) will hold ~80% of messages and its consumer CPU will saturate; the other 7 will idle. Adding 8 more consumers will not change that."

Spark: "Very skewed → one reduce bar ~90%. Salted → N extra keys, bars even, more shuffle bytes. Uniform → even bars, shuffle still non-zero."

ClickHouse: "Tenant filter on (customer_id, ts) reads << all marks. Same filter on (ts, customer_id) reads all customers in the time range."

Cardinality: "Four infra labels stay under 1e6 series. Adding user_id 1e5 crosses a TSDB budget; I will refuse that label and use CH."

If your prediction was wrong, write which assumption failed (e.g. you thought consumers split partitions).

Mapping to Grafana¶

When you later have a real cluster, recreate the sim with real panels:

• Kafka: sum by (partition) (lag)
• Spark: task duration histogram
• CH: query_log.read_rows
• Prom: prometheus_tsdb_head_series

The sim is a cartoon of those panels. The lab is a small live version. Production is the same picture with money on fire.

Classroom / pairing¶

Two people: A predicts, B drives the slider. Swap. No silent sliding. Fifteen minutes per sim is enough; then the matching lab.

Do not run more than two or three sims in a row without a module page — they will blur into "graphs go brrr."