Circuit Breakers¶

Prerequisites: Reliability Overview, Tail Latency

← Single Points of Failure | Next: Rate Limiting →

Why This Exists¶

Payments API calls Fraud. Fraud's p99 is 80ms. You set no timeout. Fraud's DB locks. Threads pile up. Payments p99 becomes "however long Fraud is sad." The fleet is latched to a dead dependency.

You add retries: 3 retries = 4 attempts, no backoff.

1000 rps inbound
Fraud times out
each request × 4 attempts  → 4000 rps at Fraud if every try is a full call
Fraud was 80% sick         → you just made it 100% dead
your thread pool           → also dead (attempts hold slots)

Timeouts, backoff, jitter, then a circuit breaker — not as decoration, as the thing that stops you from murdering a recovering service.

Abstraction Levels¶

Naive System → What Breaks¶

Unlimited retries, 30s HTTP client, one thread pool for all deps.

The Concept¶

Layer the defenses. Do not start at the breaker.

1. Timeout — every outbound call has a deadline < caller SLO. If Fraud SLO is 200ms p99, timeout 100–150ms, not 30s.
2. Retry only if safe — GET, idempotent PUT, or a request-id. Never blind-retry POST /charge.
3. Backoff + full jitter — default sleep = U(0, min(cap, base * 2^attempt)). Full jitter U(0, exp) is the default; "exp + random" (equal jitter) is a different policy.
4. Budget — retry at most X% extra load (e.g. 10%). Budget exhausted → fail.
5. Circuit breaker — closed / open / half-open on that dependency.
6. Bulkhead — separate pools / queue lengths per dependency.
7. Load shed — when we are sick, drop low-priority work cheaply (503 + Retry-After).

Breaker states:

Architecture¶

flowchart LR
    IN[Inbound 1000 rps] --> BH[Bulkhead pool]
    BH --> CB{Breaker}
    CB -->|closed / probe| T[Timeout + retry budget]
    T --> DEP[Downstream]
    CB -->|open| FAIL[Fail fast]
    DEP -->|5xx / timeout| CB
    FAIL --> SHED[Load shed / fallback]
    style CB fill:#e65100,color:#fff
    style FAIL fill:#b71c1c,color:#fff
    style DEP fill:#1565c0,color:#fff

Mechanics¶

Failure definition. Timeouts, 5xx, connection errors. Not 4xx (except 429, which is a signal to back off, not a reason to trip for everyone). Mixing 404s into the failure rate opens the breaker because of bad clients.

Window. Count (5 failures) is simple and twitchy. Percentage in a sliding window (e.g. 50% of ≥20 calls) needs a minimum throughput or a cold pod trips itself.

Open duration. 1–30s typical. Too short: you hammer the dep. Too long: you extend the outage past recovery.

Half-open. One probe per pod × 200 pods = 200 rps hammer. Global or per-cluster half-open (or a tiny probe % ) or you recreate the storm.

Fallback. Cached response, default, or "degraded UI." A fallback that hits the same DB is not a fallback.

Bulkhead sizes. pool = peak_rps × timeout_s × headroom. Fraud: 200 rps × 0.15s × 2 ≈ 60. Shipping: its own 60. Shared 60 means one dep takes them all.

Realistic Example With Numbers¶

Checkout: 1,000 rps. Fraud p50=20ms, p99=80ms. Timeout 150ms. 3 retries (4 attempts), no jitter.

Healthy:     downstream ≈ 1,000 rps     amp 1.0×
Fraud 80% timeout:
  attempts   ≈ 1 + 0.8 + 0.8² ≈ 2.44    (if all retry)
  downstream ≈ 2,400 rps                amp 2.4×
  plus 3rd retry                        → ~3,000 rps
  threads    1,000 × 0.15 × 2.4         ≈ 360 vs pool of 80 → 503s at the edge

Breaker: 50% fail over 20 calls, 5s open.

~40 ms to trip (20 calls at 1k rps is 20 ms, plus eval)
open: checkout fails fraud-check fast (~1 ms)
fallback: allow checkout < $50 on cached score; hold the rest
half-open: 1 probe / 2s / cluster, not / pod
recovery: Fraud at 200 rps of probes, not 3,000 retries

That is the difference between a 6-minute Fraud blip and a 40-minute company outage.

Interactive Explainer¶

Trip the breaker with 80% failures, then heal. Then run the retry-storm view: slow the downstream and watch amplification.

Failure Modes¶

Production Debugging¶

CPU         20% and 503s           breaker open or pool exhausted — not "need more CPU"
Memory      thread / connection leak  hung calls without timeout
Disk        usually innocent       unless the dep is your disk
Network     reset / timeout %      the failure signal you should already export
Queue depth inbound vs outbound    outbound >> inbound ⇒ retries
Lag         not Kafka — *in-flight age*
Pools       active=max, wait>0     the outage shape; dump stack, find the dep
p50/p95/p99 p50 20ms p99 3s        timeout set at 3s; cut it
Error rate  503 local vs 502 dep   open breaker vs dep still reached
Timeouts    rate ≈ fail rate       good — they are working
Retries     retry_ratio > 1.2      storm; page on this, not just 5xx
GC          pause → timeout → retry  GC caused a retry storm
Locks       dep lock + our pool     two systems, one deadlock

Dashboards that pay rent: breaker_state, calls_rejected, retry_ratio (out/in), pool_active/max, dep_p99, timeout_rate.

Scaling Limits¶

• Breakers are per dependency, per cluster — 40 microservices × 5 deps = 200 policies. Unowned policies rot to defaults.
• They do not add capacity. Open breaker = user-visible error unless you have a real fallback.
• Retry budget is a global scarce resource. Ten layers of "3 retries" is 3^10 in the worst cartoon, and 2–4× in real meshes. Cap at the edge.
• Sidecar (Envoy/Istio) breakers see sockets; app breakers see business errors. You often need both, with the same timeout story.
• Multi-tenant: one tenant's 80% errors must not open the breaker for everyone — isolate or you built a cross-tenant kill switch.

Trade-offs¶

Interview Questions¶

Reasoning Exercises¶

1. Inbound 2,000 rps, 2 retries, 30% timeout. Estimate downstream RPS and thread count if timeout=1s and you have no breaker.
2. Is retrying POST /pay after a 504 correct? What exactly must the provider guarantee?
3. Your breaker opens on 10% errors. A client bug sends 20% 401s. What happens? How do you classify failures?
4. Draw bulkheads for Checkout → {Fraud, Inventory, Tax, Email}. Email is slow; who dies if the pool is shared?

Key Takeaways¶

Previous: Single Points of Failure | Next: Rate Limiting