Technology Selection Framework¶

Design review, Tuesday 2 PM. The proposal: migrate the nightly NPS CSV job (20 MB, due Friday noon) from Airflow to Flink, "so we're stream-native before Q3." Nobody in the room can name what breaks if it stays batch.

A. Approve it — Flink is the more modern choice. B. Reject it and ask what latency requirement changed. C. Approve it, but only if two other jobs migrate too, to amortize the ops cost. D. Table it until someone writes down the SLA.

Only one of these asks the question this page exists to force. Choose tools from workload properties, not from blogs or resume pressure. This page is a procedure: questions, a mapping table, failure questions, anti-patterns, then worked examples from the running systems.

Pairwise fights live in comparisons. End-to-end boxes live in architectures. Come here first when the list of tools is still long.

The wrong question¶

"What is the best data stack?"

There is no answer. Every system in this academy is optimal for some SLAs and expensive theatre for others.

The right question¶

"What are the properties of this workload, and which tool was built for those properties? What will I refuse to add until a measurement forces it?"

Step 0 — write the consumer¶

Start by naming who waits on the result and what they do if it is late or wrong. Without that consumer, there is no stable basis for choosing a tool.

One pipeline may serve several consumers. That is several selections, not one "platform."

Step 1 — characterise the workload¶

Do this before naming Kafka.

Volume¶

• Events/s at p99 day, not the CEO's "we will be Google."
• Bytes/event (JSON vs Avro).
• GB/day uncompressed and a compression guess.
• Retention per tier (replay vs hot vs lake).

Latency (event to this consumer's answer)¶

Access pattern¶

• Point get vs scan vs multi-hop graph.
• Predictable tiles vs ad-hoc SQL.
• Join-heavy vs single-table.
• Tenant filter always present?

Cardinality¶

• Bounded labels (service, status) vs user_id / device_id.
• High card → not a TSDB. See TSDB vs OLAP.

Mutability¶

• Append-only events vs CDC upserts vs GDPR deletes.
• If you mutate, Iceberg/Hudi/OLTP — not append-only Parquet dumps.

Failure and replay¶

• Can you rebuild from Kafka (how many days)?
• Must sinks be idempotent?
• What is allowed to be dropped under overload (debug logs vs payments)?

Cost and team¶

• Who is on-call at 03:00?
• If the team has never run Flink, a minutes-SLA Spark job is not cowardice.

Write the answers in a table. If a cell is "unknown," the next task is measurement, not a PO for Pinot.

Step 2 — map to a default (then argue)¶

Ingest¶

Process¶

Store¶

Query¶

Step 3 — design for failure (or you did not select)¶

For the chosen tool, answer:

1. What metric shows it is falling behind?
2. What happens on node death mid-write?
3. What happens on a poison schema?
4. How do you replay a bad hour?
5. What is the 10× cliff (state size, parts, partitions, coordinator RAM)?

If the answer is unclear, read the matching incident before treating the choice as production-ready.

Step 4 — V1 vs zoo¶

Write fewest parts that meet the SLO. List what you will not add yet. Add a component when a named bottleneck or hard requirement appears.

See architectures index. V1 is usually Kafka + one processor + one serving store.

Anti-patterns (instant no)¶

If your doc contains three of these, stop drawing.

Worked examples (running systems)¶

Observability on-call tile¶

• Volume high, latency 1 s, access filter service+time, card high on logs, append-only, retain 7 d hot.
• Select: Kafka → Flink or CH Kafka engine → ClickHouse ORDER BY (service, timestamp). Prometheus beside for infra.
• Refuse: Pinot, Trino on the tile, ES as SoR.

E-commerce GMV¶

• Low order QPS, mutate, recon to Postgres, hours OK.
• Select: Postgres → Debezium → Kafka → Spark/dbt → Iceberg. Trino ad-hoc.
• Refuse: Flink until inventory SLA is seconds; CH as order SoR.

IoT 1-minute chart¶

• High write, latest-by-device, pyramid retain.
• Select: MQTT → Kafka → Flink windows → CH or Timescale; hourly Iceberg.
• Refuse: Prometheus for device_id; topic-per-device.

Fraud score¶

• 200 ms, keyed velocity, graph offline.
• Select: Flink/scorer + KV/CH features. Neo4j batch. Iceberg labels.
• Refuse: Spark micro-batch on the auth path; Cypher in the request.

SaaS tenant dashboard¶

• 100k/s, tenant filter, cost sensitive.
• Select: Kafka key customer_id + quotas → CH tenant-first key → Iceberg rollups.
• Refuse: per-tenant clusters in V1; notebooks as the product.

Decision record (copy this)¶

Consumer + SLO:
Volume (events/s, GB/day, retain):
Access (get / agg / join / graph):
Cardinality:
Mutability / deletes:
V1 (boxes):
Not adding yet:
Failure metric:
Replay plan:
10× change:
Anti-patterns we rejected:

If this does not fit on one page, you are designing three systems. Split the doc.

After you selected¶

1. Comparison for the pair you almost picked.
2. Architecture closest to yours — steal V1, not V3.
3. Labs for the failure you will actually hit.
4. Glossary for the words in the decision record.

Selection is finished when you can explain the choice to an on-call who does not like the tool — using the workload table, not a feature matrix.

Constraint layers (do not mix them)¶

Workload first. Org constraints second. If org forces Spark at 200 ms, the record should say SLA at risk, not "Spark is streaming."

Default V1 by consumer (summary)¶

When to reopen a decision¶

• SLO missed after ORDER BY / key design was checked (new class of engine).
• 10× volume (usually scale the same engine).
• New consumer with a different SLA (add a path, do not morph the old one).
• Team gone (ops constraint changed).

Do not reopen because a conference named a product.

Example anti-pattern write-up (copy)¶

Rejected Kafka+Flink for weekly NPS CSV (20 MB). SLA is Friday noon. Airflow + Python + Iceberg (or a sheet) meets it. Revisit if NPS becomes an in-app live widget with a 30 s SLO.

That paragraph in a design doc is the framework working.

Checklist before PR of an architecture¶

• Consumers and SLOs numbered
• GB/day math
• Keys named
• V1 boxes ≤ 5
• Not-adding list
• Failure metric per box
• Replay
• PII/retention
• Comparison pair considered
• Incident class linked

If the PR is a landscape diagram with 18 logos, bounce it.