Production Reliability Practices¶

"What breaks that we haven't found yet?"     → Chaos Engineering
"Will this survive Black Friday traffic?"     → Capacity & Load Testing
"We just had an outage — now what?"           → Blameless Postmortems

✗ "We tested that the payment service handles a database failure" —
  tested in isolation, in staging, with synthetic load.

✓ "We injected a database failure into 5% of production traffic during
  peak hours and confirmed the circuit breaker actually opened, the
  fallback actually worked, and no customer noticed" — tested in the
  one environment that has real traffic patterns, real data skew, and
  real infrastructure quirks staging never replicates.

✗ Day 1 mistake: "let's chaos-test in full production, no blast-radius limit"
  → an untested hypothesis about a poorly-understood system, at 100% blast
  radius, IS an outage you're causing yourself.

✓ Progression:
  1. Staging, full blast radius — cheapest to get wrong, learn the tooling
  2. Production, 1% of traffic, one region, business hours, team watching
  3. Production, 10% of traffic, automated abort conditions proven reliable
  4. Production, "game day" — a scheduled, larger exercise simulating a
     real dependency outage, with the on-call team responding as if real
  5. Eventually: continuous, automated, low-blast-radius chaos running
     constantly (Netflix's Chaos Monkey model) — by this point, if a
     random instance dying causes any customer-visible impact, that's
     itself the alert, and it fires long before a real, larger failure would.

Current traffic: 1,000 req/s, p99 = 150ms, everything green.

The actual question that matters: at what req/s does p99 start climbing
  non-linearly, and is that ceiling above or below what Black Friday /
  a viral moment / a marketing campaign will throw at you?

Nobody finds this out by watching a dashboard during normal traffic —
  normal traffic never reaches the ceiling. You have to go find it deliberately.

Load test finds: system degrades above 3,000 req/s.
Current peak traffic: 1,200 req/s.
Projected traffic for a marketing campaign: 3.5x current peak = 4,200 req/s.

4,200 > 3,000 → the campaign WILL push past the degradation point.
Decision: either scale infrastructure before the campaign, add caching to
reduce backend load, or explicitly accept degraded performance during the
campaign window with stakeholders informed in advance — not discovered live.

✗ Testing only the happy path — GET requests to a cached endpoint —
  while production traffic includes uncached reads, writes, and the
  occasional expensive report query. The load test's "capacity" number
  is meaningless if the traffic mix doesn't match reality.

✓ Model the realistic traffic mix: what % reads vs writes, what %
  hits cache vs misses, include the expensive/rare-but-real query
  patterns (large exports, admin dashboards) that skew p99 in production.

✗ Load testing against a scaled-down staging environment and
  extrapolating linearly — a database with 1/10th the data, on a
  smaller instance, doesn't degrade at 1/10th the load; it can degrade
  at a completely different point due to caching effects, connection
  pool sizing, or index behavior that only shows up at real data volume.

✓ Load test against production-equivalent infrastructure and, where
  possible, production-scale data — or explicitly caveat the results
  as directional, not a hard ceiling number.

✗ Blame-oriented postmortem: "Who deployed the change that caused this?"
  → the engineer who deployed it becomes defensive, downplays what they
    knew, and — more importantly — the NEXT engineer in a similar
    situation learns to hide near-misses rather than report them, because
    reporting them gets you blamed.
  → the org loses its best source of information: people who almost
    caused the SAME incident and caught it, who would otherwise have
    told you exactly where the danger was.

✓ Blameless postmortem: "What about our system, process, and tooling
  made it possible for this deploy to cause this outage, and why didn't
  we catch it sooner?"
  → the deploying engineer becomes the most valuable witness, not a
    suspect — they know exactly what the deploy process looked like
    from the inside, what warnings they did or didn't see, what they
    assumed.

✗ Same root cause (unindexed query on a schema change) causes a second
  incident 3 months later. The first postmortem's action items were
  written but never tracked to completion.

✓ Action items are tracked in the same system as other engineering work
  (the team's actual sprint board, not a separate "postmortem doc"
  nobody revisits), and a recurring incident with the same root cause
  is itself treated as a signal that the postmortem PROCESS is failing,
  not just that the system is.

Chaos engineering finds failure modes BEFORE they cause an incident
  → fewer postmortems needed, because the gap was caught in a controlled
    experiment instead of live customer traffic

Load testing finds the CAPACITY ceiling before real traffic hits it
  → fewer incidents caused by "we didn't know we'd fall over at this load"

Blameless postmortems turn the incidents that DO happen (and some always
  will — neither of the above is perfect prevention) into concrete,
  owned action items — often including "let's chaos-test this specific
  scenario going forward" or "let's load-test this before the next
  campaign," closing the loop back to the first two practices.

✗ No steady-state metric, no hypothesis, no blast radius limit —
  indistinguishable from causing an outage on purpose.
✓ Steady state, falsifiable hypothesis, bounded blast radius, automatic
  abort condition — the scientific method, applied to production.

✗ Load test result: "handles 5,000 req/s" — but only tested cached GETs.
✓ Load test with a realistic traffic mix (reads, writes, cache misses,
  the rare expensive query) — the number is only meaningful if the
  traffic shape matches production.

✗ "Root cause: engineer X forgot to add an index" — individual blame,
  discourages honest incident reporting going forward.
✓ "Root cause: our deploy process has no query-plan review step for
  schema-touching changes, and staging's data volume made this class
  of bug invisible before production" — systemic, actionable, and
  doesn't punish the person who happened to trigger a gap that existed
  before their change.

✗ "We should improve our staging environment" — a permanent wish,
  never actually done, resurfaces verbatim in the next postmortem.
✓ "@bob will add a production-scale staging dataset generator for
  high-traffic tables, due 09/05" — tracked like any other engineering
  commitment, because it is one.