Reason about data systems
the way production does.
An academy for experienced engineers. Every lesson starts from a workload, then internals, then failure. You leave able to derive an architecture — not recite a stack.
Best with basic Python and SQL · Docker is only needed for hands-on labs
00 Data systems foundations
01–02 Data representation, data movement
03–05 Distributed compute, streams, orchestration
06–09 Lakehouse, query engines, OLAP, specialised stores
10–12 Platform engineering, architecture, production
Choose a route¶
Foundations first
Scale, partitioning, and distributed execution before you touch a product name. Then Kafka and Spark.
Follow the foundations path → Most complete 02 Staff data platformDerive the whole stack
Ingestion through lakehouse, OLAP, quality, and incidents. Workload chooses the technology.
Follow the Staff path → 03 Streaming / on-callDebug lag, skew, watermarks
Kafka hot partitions, Spark stragglers, stalled Flink watermarks, ClickHouse ORDER BY mistakes.
Follow the on-call path →A curriculum built around engineering problems¶
The phases are named after systems because that is how teams talk. The lessons are named after the problem that forced the system to exist.
What breaks as volume, time, and fan-out grow — and why a log plus distributed compute is the usual answer.
Where the table lives, how analysts query it, and why dashboard latency is a storage-layout problem.
Access patterns that tabular engines lose: graphs, wide-column, time series, then catalogues, quality, and security.
Put the pieces on one diagram. Then break them. Then explain the telemetry.
Use case → why
No dictionary openings. A concrete workload, then the constraint that made the current tool fail.
Internals, not APIs
Shuffle files, ISR, watermarks, snapshots, granules. Enough to predict the next incident.
Failure makes it concrete
Gotchas, scale ladders, and incident drills connect the mental model to signals you can observe.
Same events. Different engines.
SaaS analytics, observability, e-commerce, IoT, and a fraud graph. The schema does not change; the engine does.
Sister academies: Learn ML · AI Engineering · Senior Engineer Academy