Production Debugging Playbook¶

Audience: on-call seniors | Use: war-room first 15 minutes, and interview “walk me through this outage”

You do not start with a definition of p99. You start with a page that looks almost fine.

A. Debugging High p99¶

The page: p50 = 80ms, p95 = 180ms, p99 = 6s. CPU 40%. Memory flat. Primary DB CPU 25%, no lock waits on the top query dashboard. Error rate 0.2%. Deploy was 40 minutes ago.

Most of the traffic is healthy. A thin tail is in another universe. That is not “the service is slow.” That is a subset of work taking a different path.

What do you inspect next?¶

Not “add more pods.” Not “the DB is fine so it’s the network.” Walk this tree:

flowchart TD
    S[p99 >> p95, resources look idle] --> D{Can you split the 6s by dimension?}
    D -->|tenant / shard / route / instance| H[Hot partition or sticky tenant]
    D -->|no split yet| Q{Is the request waiting or working?}
    Q -->|waiting| P[Pools, locks, queues, threads]
    Q -->|working| W[Downstream, disk, CPU in a small function]
    P --> P1[DB / Redis / HTTP pool exhausted]
    P --> P2[Lock contention / single-flight gone wrong]
    P --> P3[Executor / goroutine / Tomcat threads saturated]
    P --> P4[GC or stop-the-world]
    W --> W1[One downstream p99]
    W --> W2[DNS / TLS handshake on a subset]
    W --> W3[Packet loss → retransmit tail]
    H --> H1[One Kafka / DB partition at 20×]

First 10 minutes (order)¶

1. Confirm the window — is p99 a spike or a step change at the deploy?
2. RED + saturation — RPS, errors, pools, GC, queue depth. CPU 40% does not close anything.
3. Break the histogram — by route, instance, AZ, tenant hash, downstream span.
4. Trace 10 slow requests — not one. Look for a shared wait.
5. Mitigate before a perfect RCA: shed the hot tenant, roll back, raise the pool only if you know why it emptied.

B. Diagnosing Kafka Consumer Lag¶

The page: consumer group is Running. Members are assigned. Produce rate is normal. Committed offsets are frozen — the same offset for 12 minutes. Lag = log_end - committed only grows.

“Restart the pods” is how you turn a poison message into a rebalance storm.

flowchart TD
    L[Offsets frozen, state=Running] --> H{Heartbeat / join rate?}
    H -->|joins every few seconds| R[Rebalance storm]
    H -->|stable membership| P{Is max(partition offset) moving?}
    P -->|log end stuck too| U[Producer / leader / disk]
    P -->|log end climbing| C{Is the consumer processing?}
    C -->|busy, no commit| W[Processing slower than produce / stuck handler]
    C -->|one partition only| X[Poison message or hot partition]
    C -->|all paused, CPU 0| G[STW, deadlock, blocked on downstream]
    R --> R1[session.timeout vs max.poll.interval]
    W --> W1[slow DB, retry loop, stop-the-world GC]

What Happens in Production?¶

Same template every time: symptoms → causes → signals → diagnosis → mitigation → fix → prevention.

1. One partition at 20× traffic¶

2. Redis loses half its nodes¶

3. Replication lag 25 minutes¶

4. p50 = 90ms, p99 = 14s¶

5. Pods healthy, Service has no endpoints¶

6. Retries multiply an outage¶

7. Consumers running, offsets frozen¶

Worked example — the 6s page, minute by minute¶

You have the Module A symptoms. Here is a senior walk, not a toolbox dump.

1. Minute 0–2. Confirm it is not a scrape artifact: compare two AZs, two clients. Histogram is a cliff at ~6s, not a ramp — smells like a timeout or pool wait, not a slow algorithm.
2. Minute 2–4. Split: one route (POST /checkout/confirm) owns the tail. One downstream span, psp.authorize, is 5.9s when it is slow and 70ms when it is fast. DB spans stay 12ms. CPU 40% is now explained: threads are blocked in the client.
3. Minute 4–6. Pool: http-psp.active = 200/200, pending = 140. You did not have a PSP outage dashboard, so you thought “DB is fine.” The 0.2% errors are only the calls that exceeded 6s; everyone else is still in the pool line.
4. Mitigate. Cut checkout to a “pending payment” path; drop PSP timeout 6s → 800ms; open the circuit. p99 falls to 900ms (fail-fast) in four minutes. Conversion is hurt less than a 6s hang.
5. Fix / prevent. PSP was rate-limiting you after a retry change (scenario 6). One retry policy, pool sized to timeout × rps, alert on pool.pending > 0.

That is the standard: dimension → wait vs work → mitigate the wait → then the vendor.

Kafka: frozen offset vs slowly losing¶

Interviewers mash these together. Do not.

Commit path to recite: poll → process → commit (auto or sync). If you process and crash before commit, at-least-once redelivers. If you commit then crash mid-batch, you skip. Frozen commits with a live handler almost always means the handler never returns.

Debugging habits that survive contact with prod¶

Interview follow-ups¶

1. “CPU is only 40%, so we should scale out.” No — scale-out adds more waiters if the bottleneck is a pool or one partition. Split the tail first.
2. “Would you increase the Kafka max.poll.interval?” Only after the handler has a timeout. Raising the interval on a poison loop hides the freeze for longer.
3. “Pods are green.” Ask for Endpoints, not kubectl get pods. Green pods plus zero endpoints is a discovery outage.
4. “How do you know it is packet loss?” AZ-skewed p99, retransmit counters, app traces with no slow span. If every span is fast and the client still saw 6s, look below the process.

Self-Assessment¶

• [ ] I can name five causes of a 6s p99 with a 80ms p50 and idle CPU
• [ ] I know pool wait from downstream work on a trace
• [ ] I will not restart a Kafka group before checking per-partition lag
• [ ] I treat Endpoints: 0 as a first-class outage class
• [ ] I can explain how three retry layers multiply a blip into a brownout