Backpressure & Queueing¶

Time: 35 minutes reading + 30 minutes exercise
Prerequisites: Data at Scale, Partitioning
Outcomes: compute backlog growth from a throughput mismatch; keep queue accumulation and Little's Law in separate boxes; choose between admission control, load shedding, and autoscaling.

A producer writes 100k events/s. Kafka decouples producer and consumer rates — a topic keeps accepting writes up to its disk/retention limits rather than reacting live to a slow reader, so it does not propagate downstream slowness back to the writer the way a bounded in-process queue would. A processor downstream can only handle 70k/s. A sink after that can only accept 40k/s.

Producer  100k/s
   |
   v
 Kafka    (keeps accepting up to disk/retention limits; does not react live to consumer speed)
   |  100k/s
   v
Processor 70k/s
   |
   v
Sink      40k/s

Where does the missing 60k/s go? It does not vanish. It becomes lag — a growing backlog sitting in Kafka, in the processor's internal buffers, or in retry queues — until something either slows the producer down (backpressure) or starts dropping work (shedding). This is the same mechanism whether the "queue" is a Kafka topic, a Flink operator's input buffer, a thread pool's work queue, or an API's request queue. Learn it once here; every product-specific page (Kafka lag, Flink checkpoints, Airflow pools) is this same shape wearing a different label.

"Kafka doesn't push back on the producer" is not the same as "producers can never feel pressure." A producer can still stall or fail from:

• Local buffer exhaustion — the client-side send buffer fills and max.block.ms trips, blocking or timing out the caller.
• Broker throttling — quotas (producer_byte_rate, request quotas) deliberately slow a client down.
• Request failures/timeouts — an overloaded broker or under-replicated partition causes produce requests to fail or time out, which the client surfaces as errors, not silent success.

None of these are Kafka propagating consumer lag upstream — they are independent producer-side or broker-side limits. The distinction matters: a slow consumer, on its own, does not throttle the producer; a full producer buffer or a broker quota does.

Ask the three questions in order¶

Before any formula, decide which regime you are in. Almost every wrong answer in a lag incident comes from applying the stable-system formula to an unstable system:

1. Is the queue stable?          (is arrival ≤ service capacity?)
2. If NOT stable, how fast is    → queue accumulation:  dQ/dt = λ_in − μ
   the backlog growing?
3. If stable, what relationship  → Little's Law:        L = λW
   ties throughput, queue depth
   and waiting time together?

Question 2 and question 3 use different formulas because they are different questions. The rest of this page takes them one at a time.

Queue accumulation (not Little's Law)¶

When arrival exceeds service rate, the backlog grows linearly. This is just conservation of flow — nothing is being averaged or held constant:

dQ/dt = λ_arrival - μ_service

Arrival (λ_in)  = 100,000 events/s
Service (λ_out) =  80,000 events/s

Backlog growth  = λ_in - λ_out
                = 20,000 events/s
                = 72,000,000 events/hour

If you know current lag and the sustained arrival/service rates, you can estimate recovery time after scaling the consumer:

Current lag = 500,000,000 events
New service rate after scaling = 150,000 events/s (arrival stays 100,000/s)
Drain rate = 150,000 - 100,000 = 50,000 events/s
Recovery time ≈ 500,000,000 / 50,000 = 10,000 s ≈ 2.8 hours

This is the arithmetic behind "how long until the dashboard catches up" — a question every on-call engineer gets asked and too often answers with a guess.

Little's Law (a different question)¶

Queue accumulation answers "how big is the backlog when the system is unstable (arrival > service)?" Little's Law answers a different question: "for a stable system (arrival ≈ service, queue not growing without bound), how many items sit in the system on average, given how long each one takes?"

For any stable queue:

• \(L\) — average number of items in the system (queue depth / backlog).
• \(\lambda\) — throughput / arrival rate.
• \(W\) — average time an item spends in the system.

Example: a Flink job processes 100,000 events/s in steady state.
Each event spends 50ms in the job on average (arrival to output).

L = λW = 100,000 × 0.05 = 5,000 events in flight at any instant.

Don't reach for \(L = \lambda W\) to explain why an unstable queue (arrival > service) is growing — that's the accumulation formula above. Reach for it to size how much in-flight state (buffers, connections, in-progress records) a healthy, stable system needs to hold.

The three responses to a mismatch¶

Kafka is the middle case: it does not backpressure the producer in the classic sense (a topic keeps accepting writes up to disk/retention limits), so a slow consumer shows up as lag, not as the producer being throttled. Flink and Kafka Streams operators, by contrast, implement true backpressure — a slow sink propagates backward through the topology via bounded internal buffers, eventually slowing the source connector's read rate. Confusing these two mechanisms is why teams "fix" Kafka lag by adding consumers when the real ceiling is downstream (the sink), or "fix" a stalled Flink job by adding Kafka partitions when the real ceiling is the operator's own state size.

Levers, in the order most teams should reach for them¶

1. Fix the real bottleneck first. Measure per-stage throughput (producer, broker, each processing stage, sink) before touching any config — see Debugging below. Scaling the wrong stage just moves the queue.
2. Batch sizing. Larger batches amortize per-record overhead (network round trips, serialization) at the cost of latency. A sink batching 10k rows/write instead of 1 row/write can be the entire fix.
3. Consumer/processor autoscaling. Add parallelism — but only up to the number of Kafka partitions, and only if the new bottleneck (network, downstream sink) doesn't just move one hop over.
4. Admission control / rate limiting. Cap arrival at the edge so the system never enters the growing-backlog regime — appropriate when late data is worse than dropped data (an API, not a ledger).
5. Load shedding. Explicit, policy-driven dropping (sample, drop low-priority, drop oldest) when 1–4 aren't enough and some answer beats a fully stalled pipeline. Silent drops (buffer overflow with no metric) are the failure mode, not the fix.

How to investigate¶

Backpressure symptoms look similar across systems, but the metric that separates the hypotheses differs:

How it fails¶

• Scaling consumers when the sink, not the consumer, is the bottleneck — lag moves from "Kafka" to "processor's internal queue," which is harder to see.
• An unbounded in-memory queue "to avoid dropping data" that turns a slow sink into an OOM instead of a controlled lag metric.
• Load shedding with no metric on what was shed — silent data loss looks identical to "everything is fine" until an audit.
• Autoscaling with no maximum, so a downstream outage causes the consumer fleet to scale to a size the sink then cannot survive when it recovers (the "thundering herd on recovery" failure).
• Treating Kafka retention as backpressure — it is a deadline, not a signal. Consumer lag is not fed back to the producer, so nothing in that loop tells an independent producer to slow down (the producer-side limits above are separate mechanisms, and they do not fire because a consumer is behind).

Practice the idea¶

Open the backpressure calculator. Predict the backlog after 20 minutes with arrival at 100k/s and service at 70k/s, then test two recovery rates. The useful observation is the difference between stopping backlog growth and draining the backlog.

Check your understanding¶

An ingestion API accepts events at a sustained 100k/s during business hours. It writes to Kafka, which a Flink job consumes at 100k/s in steady state — the pipeline is healthy. A downstream ClickHouse sink hiccups and drops to 40k/s for 20 minutes before recovering.

1. Using the queue accumulation formula (not Little's Law), how much backlog (events) accumulates during the 20-minute hiccup?
2. If ClickHouse recovers to exactly 100k/s (not higher), how long does it take to drain the backlog? What does the on-call dashboard show during that time?
3. Would scaling the Flink job's parallelism help during the hiccup? Why or why not?
4. Propose one change that would have capped the backlog instead of letting it grow for the full 20 minutes.