System Design Framework¶

Prerequisites: Requirements & Estimation, HTTP APIs

← API Design | Next: Engineering Mathematics →

Why This Exists¶

Interviewer: "Design a notification system."

Most candidates draw boxes — API, Kafka, workers, FCM, Postgres — then freeze when asked "10% of devices are offline for 3 days. What happens?" The boxes never encoded retries, fan-out, or cost.

API + DB + "send loop"          → works for a demo
50M DAU Super Bowl spike?       → one table, one pool, one process
celebrity post → 10M fans?      → write amplification
device offline 3 days?          → retry storm + SMS bill
APNs slow?                      → thread pool death, p99 = 30s

Interviews test how constraints change the design, not whether you remember a reference architecture. This page is the 19-step method that forces that reasoning.

Naive System → What Breaks¶

Problem: "Send a push when someone likes your photo."

Naive: POST /notify writes a row, a cron SELECT * FROM pending and calls FCM.

The rest of this page is how each of the 19 steps changes that drawing.

The Concept¶

Nineteen steps, always in this order. Skip one and you will be surprised in production — or in the follow-up question.

Architecture¶

flowchart TD
    S1[1–4 Problem + reqs] --> S2[5 Numbers]
    S2 --> S3[6–8 API / model / v1]
    S3 --> S4[9–10 Break + scale]
    S4 --> S5[11–13 Failures + consistency + reliability]
    S5 --> S6[14–17 Observe / secure / DR / cost]
    S6 --> S7[18–19 Trade-offs + evolution]
    S4 -.->|new number| S2
    S5 -.->|new failure| S4
    style S1 fill:#1565c0,color:#fff
    style S2 fill:#6a1b9a,color:#fff
    style S3 fill:#2e7d32,color:#fff
    style S4 fill:#e65100,color:#fff
    style S5 fill:#b71c1c,color:#fff
    style S6 fill:#37474f,color:#fff
    style S7 fill:#004d40,color:#fff

Mechanics — Walkthrough: Notification System¶

Concrete product: Instagram-like likes, comments, DMs → push + in-app + optional SMS.

Steps 1–4 — Problem, questions, FRs, NFRs¶

Understand: A user wants to know something happened. They do not hire you to "run Kafka."

Ask (or die): channels (iOS/Android/web/email/SMS)? delivery SLA (chat vs like)? user preferences? mute/rate-limit? read receipts? marketing vs transactional (compliance + unsubscribe)?

Functional (in): enqueue from product events; honor prefs; at-least-once push; in-app inbox; unsubscribe.

Functional (out for v1): rich marketing campaigns, SMS two-way, read receipts.

NFRs you write down before drawing:

Step 5 — Scale (the numbers that kill v1)¶

50M DAU × 20 notifies/user/day = 1.0B/day
1.0B / 86,400 ≈ 11,600 avg QPS
peak 10× evening = 116k enqueue QPS
payload ~500 B → peak ingest 116k × 500 ≈ 58 MB/s
inbox retain 30 days → 1B × 30 × 500 B ≈ 15 TB logical
RF=3 → ~45 TB stored

If you cannot say these out loud, you cannot justify Kafka vs a table.

Steps 6–8 — API, model, simplest architecture¶

POST /v1/events            {event_id, type, actor, object, occurred_at}
                           Idempotent on event_id
GET  /v1/inbox?cursor=     user-scoped, paginated
PUT  /v1/prefs             channel × type matrix
POST /v1/devices           token, platform, user_id

v1 data: events(event_id PK), inbox(user_id, ts, event_id), prefs, devices, delivery_attempts(event_id, device, status).

v1 drawing (do this first): one write API, Postgres, one worker pool calling FCM/APNs. No Kafka. It is legal. It teaches you what to break next.

Steps 9–10 — Bottlenecks, then scale each¶

Only add a box that kills a named bottleneck. "Add Redis because interviews have Redis" is how you fail staff rounds.

Steps 11–13 — Failure, consistency, reliability¶

• Worker dies mid-send → at-least-once + event_id+device unique. Duplicate push is OK; double-SMS is not (provider idempotency key).
• APNs 503 for 12 min → circuit breaker on that channel; do not retry the world.
• Inbox replica lag 4s → chat badge may be stale; user who just sent a DM must read-your-write (session sticky to primary).
• Poison payload → DLQ after 5 attempts; never block the partition.

Steps 14–17 — Observe, secure, DR, cost¶

Observe: enqueue QPS, send QPS, queue depth, consumer lag, APNs p50/p95/p99, error rate by channel, retry amplification, pool utilization, GC pause, lock wait.

Secure: device tokens are credentials; prefs are PII; SMS is an abuse surface (rate-limit per user + per dest).

DR: RPO 0 for enqueue (multi-AZ Kafka); RTO 15 min for send path; inbox is rebuildable from events (event log is the source of truth).

Cost: APNs/FCM ≈ $0; SMS $0.0075; 1% fallback of 1B = 10M SMS = $75k/day. This single number kills "SMS fallback for everything."

Steps 18–19 — Trade-offs and evolution¶

Ship v1: Postgres + workers, likes only, no SMS. v2: Kafka + fan-out service when enqueue p99 > 200ms. v3: celebrity path when one event exceeds 100k inbox writes. Dual-write inbox schema; backfill; flip read; delete old path. No flag day.

Realistic Example With Numbers¶

A like event for a user with 400 followers:

enqueue 1 event              1 write, 0.4 ms
expand followers             400 ids from graph cache (hit 99%)
inbox batch insert           400 rows × 200 B = 80 KB
push fan-out                 400 device lookups, 2 batches to FCM
end-to-end p50               180 ms

Same event, 12M followers (celebrity):

naive fan-out-on-write       12M inbox rows = 2.4 GB, 40s of inserts
correct path                 write 1 "pointer" to celebrity timeline
                             readers merge at GET /inbox
                             push via sharded notify topics (100 partitions)
                             12M / 100 = 120k/partition, 20 workers → ~60s

The methodology is what made you split the path before the incident.

Interactive Explainer¶

Step through the 19 steps. Watch the notification design change — boxes appear only when a prior step creates pressure.

Failure Modes¶

Production Debugging¶

When "notifications are late" pages you, do not start in the FCM console.

1. CPU          worker CPU 90%? or 15% and stuck on I/O?
2. Memory / GC  old-gen climb + 800ms pauses → send p99 = pause + RTT
3. Disk         Kafka disk > 80% → produce latency, ISR shrinks
4. Network      APNs packet loss; DNS to provider
5. Queue depth  notify.priority vs notify.bulk — which is backing up?
6. Lag          consumer lag on push-workers group (msgs and seconds)
7. Pools        FCM HTTP/2 pool, DB pool, thread pool — saturation ≠ CPU
8. Latency      p50 fine, p99 12s → tail (HOL, slow 1% dep), not "the app"
9. Error rate   429 from APNs vs 5xx from us — different levers
10. Timeouts    client 3s, worker 10s, FCM 30s → retry amplification
11. Retries     downstream RPS / inbound RPS > 1.5 → storm
12. Locks       inbox hot-row updates for unread_count

Scaling Limits¶

• One inbox table dies around tens of k writes/s — shard by user_id before 50k.
• One Kafka partition is one consumer — celebrity push needs many partitions.
• APNs/FCM have their own quotas; your 116k QPS is not their 116k QPS.
• Fan-out-on-write storage grows as events × avg_followers — celebrities make this super-linear.
• Multi-region send is easy; multi-region prefs + unread is a consistency product.

Trade-offs¶

Interview Questions¶

Reasoning Exercises¶

1. Re-run steps 5, 9, 10, 17 for chat (WhatsApp-style, p99 400ms) instead of likes. Which boxes disappear? Which become mandatory?
2. You are at step 8 with 10 engineers and 6 weeks. Which three steps do you still refuse to skip, and why?
3. Marketing wants the same pipeline. Walk steps 15 and 17. What new entities appear (consent, frequency cap, suppression list)?
4. Inbox is in Postgres. Step 19: migrate to a wide-column store. Write the dual-write / backfill / flip order and the consistency hole you will accept.

Key Takeaways¶

Previous: API Design | Next: Engineering Mathematics