Skip to content

Learning paths

Do not try to read every page in order. Pick a route with a finish line. A reading session can be 25–40 minutes; a lab can be a separate session. Optional depth never blocks progress.

How to use a path

On the first visit, read the opening and Build the mental picture, then attempt the exit criterion. Return for internals and operations when the route calls for them. Use the three-pass rhythm instead of forcing every page into one sitting.

A gentle first hour

If every route still looks large, begin here:

  1. Read the story and mental picture in Data at Scale.
  2. Open the Kafka partition simulator. Keep the defaults, then make one key receive 80% of events.
  3. Read only the opening and mental picture in Partitioning.
  4. Finish this sentence in your own words: “We partition data because …, but a poor key causes …”.

That is a complete session. The next session starts with distributed execution; no Docker or product-specific knowledge is required yet.

Core path — six weeks

This is the shortest complete route. Check off evidence, not page views.

Week Required sessions Deliverable
1 Scale, partitioning, data modelling Workload table + grains/keys
2 Kafka log/partitions, CDC Produce/consume lab + CDC handoff
3 Spark mental model/shuffle, transformations Spark UI screenshot + incremental model
4 Flink time/state, Airflow idempotency Watermark model + replay plan
5 Lakehouse, Trino/warehouse, ClickHouse Storage/query decision record
6 Quality, security, cost, architecture Capstone scored ≥15/18

Progress evidence:

  • Quantified requirements
  • Declared grains, keys, and time semantics
  • Replay and reconciliation plan
  • Lab evidence captured
  • Failure runbook exercised
  • Unit-cost worksheet
  • Capstone reviewed at 15/18 or better

Foundations first

Who: senior engineers who have used Spark or Kafka as a library and have never had to explain a shuffle or an ISR.

Order Page Exit criterion
1 Start here You can state the academy objective in your own words
2 Data at scale You can say what breaks at 100 GB vs 10 TB vs 1 PB/day
3 Partitioning You can design a key and name the skew failure
4 Data modelling Every fact has a grain and history policy
5 Distributed execution Job → stage → task is automatic
6 The log You can explain why a log is not a queue and not a database
7 CDC Snapshot and stream meet without lost updates
8 The shuffle You can draw shuffle write/read and a straggler
9 Transformation engineering Incremental equals full rebuild
10 Batch vs stream You can pick one for a given latency and correctness need

Then continue into Kafka partitions and Spark gotchas.

Staff data platform

Who: people who will be asked to own the stack, not a job.

Follow Foundations first, then:

Order Page Why it is on the path
1 Kafka replication Durability is a replica and ack story
2 Exactly-once EOS is a protocol, not a checkbox
3 Iceberg A table on object storage is metadata
4 Trino Federation and why the coordinator dies
5 ClickHouse Dashboard latency is physical design
6 Airflow idempotency Orchestration is not processing
7 Quality Green jobs, wrong numbers
8 Cost engineering Unit cost, not just uptime, is what you own
9 Platform delivery CI/CD and IaC for a data platform, not a webapp
10 SaaS analytics architecture Put it on one diagram
11 Selection framework Workload chooses the tool

On-call streaming

Who: you get paged for lag, stalled jobs, or “the dashboard is empty.”

Order Page Incident it prepares
1 Kafka partitions Hot partition, consumer lag on one p
2 Kafka gotchas Rebalance storms, poison pills
3 Spark gotchas Executor OOM, skew join
4 Time semantics “Why did the window never close?”
5 Watermarks Idle source, late events
6 Checkpoints Recovery that replays too far
7 Incidents Full drills with telemetry

Run the Kafka, Spark, and Flink labs. Do not skip the “now break it” steps.

Lakehouse and analytics

Who: warehouse / lake migrations, “should we use Iceberg or Delta,” dashboard SLAs.

Only ClickHouse has a hands-on lab here

Iceberg, Hudi, Delta, Trino, and Pinot have no Compose lab in labs/ yet — for those, this path is reading plus simulations only. Run the ClickHouse lab for hands-on time in this path.

  1. Why table formats exist
  2. Iceberg, Hudi, Delta
  3. Choosing a table format
  4. Columnar storage
  5. ClickHouse vs Pinot vs Trino
  6. ClickHouse vs Pinot, ClickHouse vs Trino
  7. Observability architecture or SaaS analytics

Interview / design sprint (two weeks)

Not an interview-prep clone. Use this if you need to talk about data platforms in a Staff interview quickly.

Week 1: Scale, Partitions, Shuffle, Log, Replication, Event time.

Week 2: Iceberg, ClickHouse, Selection framework, then one architecture page covered and re-derived: Observability or Fraud.

Protocol: cover the solution, write requirements, V1, bottleneck, V2, failure modes. Never start from the finished diagram.

Specialised stores

Jump here only after you can explain why a warehouse is the wrong tool for the access pattern.