Flink vs Kafka Streams vs Spark Structured Streaming¶

In tomorrow's design review, the fraud team proposes routing "10 failed logins in 5 minutes" through Spark Structured Streaming, because the platform team already runs a Spark cluster for the lake. The ticket's SLO says alert within 2 seconds of the 11^th failure.

Before you read the comparison table below, predict: does Spark Structured Streaming meet that SLO?

A. Yes — Spark is Spark, it will just run faster with more executors. B. No — micro-batch triggers put realistic latency at seconds to minutes, not sub-2-second, regardless of cluster size. C. Only if you enable continuous processing mode, which is production-ready. D. It depends only on how the DataFrame is partitioned.

Start with the situation¶

You already have {timestamp, customer_id, user_id, service, endpoint, region, latency_ms, status_code, bytes} in Kafka. A team asks "should this be Flink?" The staff-engineer answer is never the vendor matrix first. It is: what is the workload, the state, the latency, and who will be paged?

This page compares three engines that all can window a Kafka topic. It does not declare a winner.

Why the obvious approach breaks at scale¶

All three products advertise event time, exactly-once, and Kafka. The differences show up when:

• State is 80 GB, not 80 MB
• You need 200 ms p99, not 30 s
• You need to join a stream with a slowly changing table
• The team is Spark-native or Java-microservice-native
• The sink is ClickHouse, not another topic

A feature table will say "yes, yes, yes". Production will not.

Intuition: three deployment philosophies¶

Flink — a cluster whose job is to run dataflows. You scale TaskManagers independently of the apps that produce events. State lives on those TMs (RocksDB) and in checkpoints on object storage. Best mental model: "stream processor as infrastructure".

Kafka Streams — a library inside your JVM service. Partitions of the input topic are the parallelism (one task per partition, roughly). State is local RocksDB plus changelog topics. Best mental model: "the microservice is the processor".

Spark Structured Streaming — micro-batches (or a continuous mode you will rarely ship) on a Spark cluster. You reuse DataFrames, Catalyst, and the lakehouse sinks you already run at 02:00. Best mental model: "the batch engine, ticking".

flowchart TB
    subgraph flink [Flink]
      JM[JobManager]
      TM[TaskManagers + RocksDB]
      JM --> TM
    end
    subgraph ks [Kafka Streams]
      P1[Pod 1 + RocksDB]
      P2[Pod 2 + RocksDB]
    end
    subgraph spark [Spark SS]
      D[Driver]
      E[Executors]
      D --> E
    end
    K[(Kafka)] --> TM
    K --> P1
    K --> P2
    K --> E

Internals: what actually has to run¶

Flink: JobManager + TaskManagers, checkpoint storage, a scheduler. Failure unit: TM or JM.

Kafka Streams: instances in a consumer group, local RocksDB, changelog topics, group coordinator. Failure unit: the pod, via rebalance.

Spark SS: driver + executors, checkpoint dir of offsets + sink commits, a trigger loop. Failure unit: the micro-batch.

All three read Kafka partitions; none of them add parallelism beyond what keys/partitions allow without a shuffle. Watermarks (Flink/Spark) and grace periods (Streams) are the same late-data idea with different knobs.

Mental models (keep these)¶

Flink¶

Separate cluster (K8s, YARN, standalone). First-class event time, watermarks, idle sources, large keyed state, CEP. Operational cost: JobManagers, checkpoints, savepoints, UI. See time, state, checkpoints.

Kafka Streams¶

No JobManager. Scale by adding app instances; group membership maps partitions (Kafka consumers). Rebalances are your failover. State restore from changelog can dominate a bounce. Interactive queries (query local state) are a unique fit for "this service needs the count now".

Spark Structured Streaming¶

Each trigger processes the new Kafka offsets as a DataFrame. Watermark + groupBy(window). Latency typically seconds to minutes. Exactly-once to Iceberg/Delta is excellent. Exactly-once to a low-latency Kafka topic is not why you bought Spark.

Comparison by dimension (secondary)¶

Use this only after the workloads below.

Workload 1 — Fraud: 10 failed logins in 5 minutes¶

Requirements: event time, 5-minute sliding window, p99 < 2s to alert, state = active users, Kafka in/out.

Pick: Flink if a platform team owns streaming; Kafka Streams if a single Java service owns the rule and state is modest.

Workload 2 — Observability: p95 latency per service per minute at 2M events/s¶

Requirements: huge ingest, cheap aggregations, late events, sink to Pinot/ClickHouse, loss of a few lates acceptable.

Pick: often Spark SS or OLAP ingest, not Kafka Streams. Flink when freshness must be ~second and you already run Flink.

Workload 3 — SaaS analytics dashboard: error rate per customer_id per minute¶

Requirements: tenant skew (whales), event time, sink to a serving store, 1–5s freshness.

Pick: Flink or Kafka Streams for "live"; Spark if the dashboard already polls a warehouse.

Workload 4 — E-commerce: enrich clicks with user profile, sessionise¶

Requirements: stream-table join, session windows, moderate rate, JVM commerce services.

Pick: Kafka Streams unless state/join complexity outgrows the library.

Workload 5 — IoT: 20M devices, sparse traffic, "offline if no event in 15 minutes"¶

Requirements: idle partitions, processing-time or event-time timers, huge key cardinality, TTL.

Pick: Flink with aggressive TTL; or a purpose-built TSDB/heartbeat store.

Workload 6 — Lake ETL: clean Kafka into Iceberg every minute¶

Requirements: schema evolution, exactly-once table commits, SQL, same job as nightly backfill.

Internals recap (latency budget)¶

Fraud 200 ms: you cannot wait for a Spark trigger or a 60s Flink checkpoint to emit (checkpoints are for recovery, not for the alert path). The alert record leaves the window operator when the watermark says so; EOS only delays committed Kafka visibility by the checkpoint interval if you used transactional sinks. For fraud, many teams use at-least-once sink + idempotent paging.

How: a decision sequence¶

Is the sink a lakehouse table and latency ≥ 15s?
  → Spark Structured Streaming (or batch every N minutes)

Is the processor a single JVM microservice, state < tens of GB,
and parallelism = Kafka partitions?
  → Kafka Streams

Do you need CEP, huge state, independent scaling, or mixed
sources with serious watermarks/idleness?
  → Flink

Is it "10 failed logins in 5 minutes" and you have no stream
platform yet?
  → Start with Kafka Streams or Flink SQL on a small cluster;
     do not start with Spark if the SLO is seconds.

Gotchas when teams "just pick Flink"¶

• PyFlink at 2M/s hot paths: you will rewrite operators in Java.
• Kafka Streams + huge state + Kubernetes preStop 10s: restore from changelog exceeds the budget; use standby replicas.
• Spark SS "continuous processing": still not the default; do not design fraud on it.
• Exactly-once still does not include Stripe. Same lecture as Kafka EOS.

Failure modes (by engine)¶

Debugging metrics: Flink checkpoint duration + watermark; Kafka Streams records-lag-max + rebalance; Spark latestBatchProcessingTime + Kafka lag.

Scale: 10× / 100× / 1000׶

Trade-offs¶

You are trading operational surface (Flink/Spark clusters) against coupling (Kafka Streams inside the app) against latency (Spark micro-batch).

Alternatives¶

• ksqlDB — SQL on Kafka Streams; good for simple filters/joins, less for heavy state.
• ClickHouse / Pinot materialized from Kafka — skip the processor for observability-shaped jobs.
• Batch only — if "5 minutes" can be a 5-minute Airflow DAG. See batch vs stream.

Cost and people (the dimension vendors skip)¶

Flink: a platform team, 24/7 cluster, checkpoint storage, a UI people must learn. Worth it if several jobs share that platform.

Kafka Streams: no cluster, but every microservice becomes a stateful stream processor — disk, restore, rebalances, changelog topics. Worth it if that team already ships JVM services and the job dies when the service dies (acceptable).

Spark SS: you already pay for Spark. Incremental cost is a long-running job stealing executors from batch. Worth it if latency is batch-shaped and the sink is the lake.

Headcount is an architectural constraint. A two-person data team should not run all three.

Migration sketches¶

Kafka Streams → Flink when changelog restore exceeds deploy budgets or you need idle watermarks and CEP. Plan key compatibility and a dual-run on a new output topic.

Spark SS → Flink when product moves a dashboard from 1 minute to 1 second and the extra second is revenue. Do not migrate lake ETL.

Flink → Spark SS when the job is actually a 30s rollup into Iceberg and you are tired of operating RocksDB. This happens more often than conference talks admit.

Always keep Kafka as the system of record for the stream. Processors are replaceable; topics with a documented schema are not.

A fourth option, not in the title: do the aggregation in the serving store. Pinot/ClickHouse consuming Kafka with a 1-minute rollup removes Flink from observability more often than it removes Spark from the lake. Evaluate it as a first-class alternative on workload 2, not as an afterthought.

If the organisation already standardised on one engine, still run this workload list. Standardising on Spark does not make fraud fire in 200 ms. Standardising on Flink does not make Iceberg writers cheaper. Standards are defaults, not physics.

Debugging across engines (same Kafka topic)¶

The input is always Kafka. Start there: consumer lag by partition, URP, ingest rate. Then branch:

Do not tune Flink parallelism because a Kafka partition is hot. The engines do not repeal partition math.

How to apply this at work¶

Write the workload in one paragraph (latency, state size, sink, team). Map it to one of the six workloads above. If it maps to none, you do not understand the workload yet — do not pick an engine to postpone that.

Check your understanding¶

A company has Spark for the lake, Kafka for ingest, and a Java order service. They want (a) Iceberg tables from service-events within 1 minute, (b) sessionisation of shoppers for a recommendation service with 200 ms reads of "current session", © the fraud login rule at < 1s.

Which engine for a, b, c — and what do you not unify?