Reference¶
A design review: someone asks "why not Databricks for everything?" A teammate opens this page, searches for "selection framework," and starts reading glossary definitions instead of answering the question.
Predict before you read on: is that the right reflex, or is reference the wrong place to start when you are still building intuition?
It's the wrong place. Use this section when you need a precise term or a version pin, not a tutorial or a decision procedure — Versions & Primary Sources before applying version-sensitive examples in production, the Glossary for an operational term you hit mid-page. If you are still building intuition, go back to the module (foundations → tool → architecture); looking up "watermark" every time is a signal to replay Flink time, not to live in the glossary.
Selection framework and cost engineering moved
The technology selection framework and cost engineering are now taught as lessons in Phase 11: Architecture & Economics under Learn, not looked up here as reference material — they are decision procedures you work through with a real workload, not definitions you check mid-page. This page keeps the links below for continuity; the module pages themselves are unchanged.
Contents¶
| Page | Use it when |
|---|---|
| Technology selection framework | Designing a system, defending a choice, killing a zoo — now Phase 11 |
| Cost engineering | Converting architecture into unit economics and a budget — now Phase 11 |
| Versions and primary sources | Checking lab pins and version-sensitive behavior |
| Glossary | Operational meaning of a word used in this academy |
When to open the selection framework¶
- A stakeholder asks "why not Databricks / Flink / ClickHouse for everything?"
- You are about to add Kafka "because we might stream later."
- Two teams want Spark and Flink and Ray for the same job.
- You have a workload (events/s, SLA, query shape) and two names left — after a comparison.
The framework is a question list plus anti-patterns. It will not pick a vendor SKU. It will stop you from buying Pinot for 5 QPS.
When to open the glossary¶
- A page said ISR, granule, watermark, manifest and you need the ops definition.
- You are writing a design doc and do not want Wikipedia's definition of "exactly-once."
- You are explaining a term to someone and want the academy's meaning (which includes why it pages you).
The glossary is 40–60 terms that show up in this repo. It is not a data-engineering encyclopedia.
How reference relates to the rest¶
flowchart TD
W[Workload: volume latency access retention cost failure] --> F[Selection framework]
F --> C[Comparison pages]
C --> A[Architecture V1 then V2]
A --> L[Labs]
L --> I[Incidents]
I --> G[Glossary for the words in the write-up]If you start at the glossary, you will collect definitions. If you start at the workload, the glossary is a checksum.
Fast pointers (do not substitute for modules)¶
| You need | Go here |
|---|---|
| Batch vs stream | foundations, Spark vs Flink |
| Partitioning | foundations/partitions, Kafka/Spark labs |
| Lake vs OLAP vs TSDB | CH vs Trino, TSDB vs OLAP |
| Graph vs SQL | graph vs relational |
| Operating the platform | metadata, quality, security |
| End-to-end boxes | architectures |
What this section is not¶
- Not a certification dump.
- Not a replacement for incidents (those train judgement).
- Not vendor pricing.
- Not "best practices" detached from a workload.
Suggested 30-minute use¶
- Pick a real system at work (even a modest one).
- Fill the framework's workload table with numbers (guesses OK if labelled).
- Write choose-X / choose-Y / choose-neither for one pair.
- Check glossary terms you used in the write-up — if you cannot define lag operationally, fix that before the design review.
Contributing to reference¶
Add a glossary term when a module uses it and the ops meaning is easy to get wrong (e.g. exactly-once vs idempotent sink). Do not add terms that are not in the academy.
Add a framework anti-pattern when you see it twice in real design docs. One-off vendor quirks stay in module gotchas.
Cheat sheet: tool → first metric¶
| Tool | First thing to graph |
|---|---|
| Kafka | Lag per partition, ISR, disk |
| Spark | Stage max/median task time, shuffle read |
| Flink | Watermark vs now, checkpoint duration, backpressure |
| ClickHouse | Parts, query_log marks, insert delay |
| Iceberg | Snapshot count, files per partition, plan time |
| Trino | Coordinator heap, bytes scanned, broadcast vs partitioned |
| Prometheus | Head series, scrape duration |
If you cannot name the metric, you are not done selecting the tool. Details: glossary, incidents.
Cheat sheet: word → module¶
| Word | Module |
|---|---|
| Watermark | Flink time |
| Shuffle | Spark shuffle |
| Granule / ORDER BY | ClickHouse |
| Snapshot / manifest | Iceberg |
| Cardinality | time series |
| ISR / ack | Kafka replication |
| Idempotent DAG | Airflow |
How not to use this section¶
Do not quote the framework at a product manager without the workload table filled. "We chose ClickHouse because the framework said dashboards <100 ms" is valid only if you wrote the query. Empty framework checkboxes are theatre.
Do not dump the glossary into onboarding as 60 flashcards. Assign start here and let terms appear in labs.
Link map¶
- Comparisons: index
- Architectures: index
- Labs: index
- Simulations: index
- Platform: metadata, quality, security, notebooks
Decision record archive (personal)¶
Keep a folder of one-pagers (the framework template). After six months you will see the same anti-patterns. That archive is more valuable than re-reading this index. This section is the legend for those pages: terms, metrics, links.
Common lookups¶
"Do I need Flink?" — framework latency row, then Spark vs Flink.
"Hot key" — glossary hot partition, incident 1, Kafka lab.
"Why is the dashboard slow?" — CH ORDER BY / parts, incident 4, CH vs Trino if someone pointed Grafana at Trino.
"user_id in Prometheus" — glossary cardinality, TSDB vs OLAP.
"Can Trino replace CH?" — No for 100 ms tiles. Yes for ad-hoc lake. Framework + comparison.
Index of anti-patterns (short)¶
Kafka-for-someday, Flink-because-Kafka, lake-as-OLTP, Trino-for-Grafana, Prom-high-card, Neo4j-on-auth, notebooks-as-DAG, dual-OLAP-in-V1, Ray-for-SQL, one-cluster-all-SLAs, schema-less-JSON, shard-later-without-keys.
Full prose: selection framework.
How terms are chosen for the glossary¶
A term lands in glossary if:
- It appears in multiple academy pages, and
- The Wikipedia definition misses the ops consequence (e.g. exactly-once vs idempotent sink).
If you need a term that fails (2), it stays in the module.
10-minute drill¶
Open the framework. Fill volume/latency/access for your last project. Circle one anti-pattern you committed. Write the V1 you would ship now. That is the use of this section. Re-reading definitions without a workload is studying for a quiz, not for on-call.