Tail Latency¶

Prerequisites: Caching & Performance, Capacity Estimation

← Cache Strategies | Next: Cache Stampede →

Why This Exists¶

Dashboard: CPU 40%, avg 95ms, error rate 0.1%. Slack is on fire. Checkout "takes forever."

You open percentiles:

p50 = 80ms     ← the average's friend; most users are fine
p95 = 180ms    ← still inside the 200ms SLO
p99 = 6s       ← 1 in 100 checkouts is a lost session

A 1k rps service produces 10 requests/s that wait 6 seconds. That is 36,000 terrible checkouts per hour. Averages hid an outage because they are pulled by the 99, not named by them.

If 2% of users abandon at 3s, that tail is not a graph — it is revenue. The SLO belongs on p99, not on mean.

Naive System → What Breaks¶

You SLO on average latency and scale when CPU > 70%.

The Concept¶

Percentile pN = the latency that N% of requests beat. p99 = 99% faster than this; 1% slower.

Why tails form:

1. Queueing — when utilization ρ → 1, wait time explodes (not linearly).
2. Little's Law — L = λW (in-flight = arrival rate × time in system). Slow W inflates L; finite pools then inflate W. Spiral.
3. Head-of-line (HOL) blocking — one fat/slow request occupies a worker, connection, or HTTP/1.1 socket; the thin requests behind it inherit its wait.
4. Pool exhaustion — DB pool, GC threads, connection pool: waiters sit in a queue the CPU dashboard cannot see.
5. Fan-out — a page that calls N backends inherits the max of their tails.

The 40% CPU, 6s p99 pattern: you are not compute-bound. You are concurrency-bound. The missing 60% CPU is threads stuck on a lock, a disk, or a 1% dependency.

Architecture¶

flowchart LR
    REQ[Requests λ] --> Q[Run queue]
    Q --> W[Workers / pool]
    W --> DEP[Deps + disk + lock]
    DEP -->|slow 1%| TAIL[p99 / p999]
    W -->|fast 99%| OK[p50]
    Q -.->|HOL| TAIL
    style TAIL fill:#b71c1c,color:#fff
    style OK fill:#2e7d32,color:#fff
    style Q fill:#e65100,color:#fff

Mechanics¶

Queueing in one line¶

For an M/M/1-ish server, E[wait] ≈ ρ / (1−ρ) × S. At 50% busy, wait ≈ service. At 80%, wait = 4×. At 95%, wait = 19×. p99 is much worse than the mean wait. Headroom is a latency feature.

Little's Law, used on-call¶

λ = 1,000 rps    W = 80ms = 0.08s    L ≈ 80 in-flight     — healthy
λ = 1,000 rps    W = 6s              L ≈ 6,000            — you do not have 6,000 workers

If the pool is 200, 5,800 requests sit in an accept/queue. Their latency is queue time, not handler time. CPU can be 40% (200 workers blocked on I/O).

HOL blocking¶

HTTP/1.1 one-connection-per-client, a single JDBC connection reused serially, a single-threaded event loop doing a 200ms JSON parse, a Kafka partition whose consumer hits a poison 10s call — all HOL. The fix is isolation: HTTP/2 multiplexing (with care), separate pools for fast/slow, hedged requests, or cancel.

Slow 1% dependency¶

page = auth + recs + ads + inventory + price
each p99 = 200ms independently
P(page > 200ms) ≈ 1 − 0.99^5 ≈ 5%

You are already at p95 of the page from five "p99=200ms" toys. This is why edge SLOs cannot equal the worst backend SLO.

Hedge / tail chopping (use sparingly)¶

If p99 is 10× p50, a hedged retry after p95 wait can cut tail — and double load on the tail-causing dep. Pair with retry budgets.

Coordinated omission¶

A closed-loop load test waits for each response before sending the next. When the system slows, the generator slows — p99 looks better than production, because the waits never entered the sample. Open-loop generators (constant λ) and measuring intended start time expose the real tail. If your load test p99 is 180ms and prod p99 is 6s at the same CPU, believe prod and suspect the test.

What to chart together¶

Always plot p50 / p95 / p99 / p99.9 + avg on one graph, plus in_flight, pool_wait_ms, and CPU. The signature avg≈p50≪p99, CPU mid, pool_wait≈p99 is this entire page in one screenshot.

Realistic Example With Numbers¶

Checkout API: 800 rps, 32 Java workers, DB pool 20, CPU 40%.

p50  80ms     handler 50ms + DB 30ms
p95  180ms    cache miss + lock wait
p99  6s       DB pool wait 5.8s + 200ms query
avg  140ms    looks "a bit slow"

What happened: a new query on orders full-scans for 1% of users (big accounts). Those hold a pool connection 2s. 800 rps × 1% = 8 slow/s. 8 × 2s = 16 extra connections — almost the whole pool of 20. Everyone else queues. The 99% pay for the 1%.

Fix order: index / kill the query (cause), then timeout the query at 100ms (bound), then split pools (isolate), then more connections (last — they amplify lock contention).

Interactive Explainer¶

Run a healthy stream, then inject HOL blocking or a 1% slow dependency. Watch avg stay polite while p99 detaches.

Failure Modes¶

Production Debugging¶

CPU         40% + p99 6s         not a scale-out; find blockers
Memory      swap / GC old-gen    pauses become p99
Disk        fsync / read amp     one slow volume poisons a shard
Network     retrans, DNS         1% of calls take 5s (nscd / IPv6)
Queue depth run queue, LB, Kafka wait time is latency you own
Lag         consumer / replica   users read stale *and* wait
Pools       db, http, threads    active=max, wait_ms ≈ p99  → smoking gun
p50/p95/p99 always chart together if p99/p50 > 10, you have a tail problem
Error rate  may be *low*         timeouts not counted? they are the tail
Timeouts    client 3s, you 30s   user sees 3s; you still hold the pool
Retries     amplify the 1%       1% slow → 3% load on the slow path
GC          match pause to p99   if equal, the tail *is* the collector
Locks       one key / one row    celebrity user, global mutex, flights table

Trace one p99 request. If 5.8s is pool.acquire, you are done hunting the handler.

Scaling Limits¶

• You cannot SLO p100. Hardware, GC, and the kernel have a max. SLO p99 or p99.9 and budget error + timeout.
• Fan-out of 50 serial deps: p99 becomes fiction. Collapse deps, parallelize with a deadline, or cache.
• Coordinated omission: if the load generator waits for a response before sending the next, it understates p99. Use a closed vs open model on purpose.
• Multi-tenant: one tenant's 1% is everyone's p99 on a shared pool — isolate or you will "optimize" the wrong customer.
• p99 of a 10 rps service is statistically noisy (1 sample / 10s). Collect longer windows or use p95.

Trade-offs¶

Interview Questions¶

Reasoning Exercises¶

1. 500 rps, mean service 10ms, one worker. At what ρ does mean wait exceed 50ms (M/M/1)? What happens to p99 sooner than that?
2. DB pool 10. 2% of queries take 1s, the rest 10ms, 200 rps. Estimate connections held by the slow class. Who queues?
3. You add 3 retries with a 2s timeout to "fix" p99. What happens to L and to the dependency? Sketch the worse p99.
4. Two tenants share a pool. Tenant A is 1% of QPS and 90% of hold time. Propose isolation that does not require a rewrite.

Key Takeaways¶

Previous: Cache Strategies | Next: Cache Stampede