Design: Recommendation System (e.g. "Videos for you" / "Products you may like")¶

Difficulty: Senior/Staff | Time: 60–75 minutes

1. Problem Statement¶

Design the system behind a "Videos for you" or "Products you may like" surface: given a user (and maybe a context — what they're currently viewing), return a small, personalized, ranked list of items from a catalog of millions to billions.

The naive framing — "score every item for every user, return the top K" — is intractable, and saying so precisely is the first thing an interviewer listens for. If you have 500M users and a 50M-item catalog, and each user×item scoring call costs even 1ms on a capable model, a full scan is 500M × 50M × 1ms — not a number you can make up with more machines. You cannot rank everything for every user on every request.

The industry-standard answer is a two-stage funnel:

1. Candidate generation — cheaply narrow billions of items down to a few hundred plausible candidates for this user, using lightweight methods (embeddings + approximate nearest-neighbor search, popularity, co-occurrence). Optimizes for recall: don't miss good items, false positives are fine because ranking will filter them.
2. Ranking — expensively and precisely score those few hundred candidates with a heavier model that can use rich features (user history, item metadata, context, cross-features), and order them. Optimizes for precision on a small set, because that's the only place you can afford to spend real compute per request.

This split is the entire architecture. Every version below is either improving candidate generation, improving ranking, or improving the freshness of the data both stages read from. For the mechanics of hosting the ranking model itself — batching, GPU autoscaling, cold starts — see Model Serving; this page is about what candidates to feed it and how to score them for this problem, not general inference infrastructure.

2. Clarifying Questions¶

3. Functional Requirements¶

• Return a ranked list of N personalized items for a given user + optional context (e.g. "currently viewing item X")
• Log user interaction/feedback events (impression, click, watch/purchase, dwell time) as they happen
• Exclude items the user has already interacted with (configurable — "seen" filtering) and items ineligible for the user (region-locked, out of stock)
• Support both a "for you" homepage surface and a "related items" contextual surface from the same underlying pipeline
• Recommendations must incorporate a user's very recent activity, not just historical behavior

4. Non-Functional Requirements¶

5. Capacity Estimation¶

Users & catalog:
  200M DAU, 500M total users
  Item catalog: 50M items (video/product scale), growing ~200K new items/day

Requests:
  200M DAU × 8 recommendation surface loads/day ≈ 1.6B requests/day
  ≈ 18,500 rps average, ~5x peak ≈ 92,500 rps

Candidate generation (per request):
  ANN search over embedding index: top ~500 candidates from a few sources
    (embedding similarity ~300, popularity fallback ~100, recently-viewed-similar ~100)
  ANN query latency budget: single-digit ms per source, run in parallel

Ranking (per request):
  ~300-500 deduplicated candidates scored by the ranking model
  ~50-150 features per (user, item) pair: user embedding, item embedding,
    recency features, cross features (user-category affinity), context features
  92,500 rps × 400 candidates ≈ 37M scoring calls/second at peak
    — this number is why ranking must be a BATCHED model-serving problem
      (see Model Serving) and why it only ever runs on the narrowed
      candidate set, never the full 50M-item catalog

Event log (training data source):
  1.6B requests/day × ~10 impressions shown ≈ 16B impression events/day
  Plus click/watch/purchase events, maybe 5-10% of impressions ≈ 1-1.5B/day
  At ~200 bytes/event ≈ 3.5TB/day raw event volume (before compaction/aggregation)

Embedding storage:
  User embeddings: 200M users × 128 dims × 4 bytes ≈ 102GB
  Item embeddings: 50M items × 128 dims × 4 bytes ≈ 25GB
  Both must be servable at low latency — this is a "fits in a fast KV/vector
  store" budget, not a "scan a table" budget

6. API Design¶

GET /v1/recommendations
  ?user_id=...&context=home|item_detail&context_item_id=...&limit=20
  → {
      items: [{ item_id, score, reason_code }],
      request_id            # for feedback correlation
    }

POST /v1/events
  {
    request_id,             # ties feedback back to the serving request that showed it
    user_id, item_id,
    event_type: "impression" | "click" | "watch" | "purchase" | "dismiss",
    dwell_ms?,
    ts
  }
  → 202 Accepted   # fire-and-forget, never on the critical UX path

# Internal, not client-facing
GET  /internal/embeddings/user/{user_id}
GET  /internal/embeddings/item/{item_id}
POST /internal/reindex   # trigger ANN index rebuild/merge (admin)

request_id on the recommend call and its echo on the feedback event is what makes the interaction log usable as labeled training data later — without it you have impressions and clicks in two streams with no reliable way to join them at the granularity a ranking model needs.

7. Data Model¶

-- User features: durable profile signal, refreshed by the offline pipeline.
-- NOT the hot serving path by itself — see the online feature store below.
CREATE TABLE user_features (
    user_id        UUID PRIMARY KEY,
    embedding      VECTOR(128),         -- learned representation
    top_categories JSONB,               -- affinity summary, small
    updated_at     TIMESTAMPTZ NOT NULL
);

-- Item features: catalog metadata + learned embedding.
CREATE TABLE item_features (
    item_id        UUID PRIMARY KEY,
    embedding      VECTOR(128),
    category       VARCHAR(64),
    popularity_7d  BIGINT,
    created_at     TIMESTAMPTZ NOT NULL,
    is_active      BOOLEAN DEFAULT TRUE
);

-- Interaction/event log: append-only, the SOURCE of training data.
-- Partitioned by day, never updated in place.
CREATE TABLE interaction_events (
    request_id     UUID NOT NULL,
    user_id        UUID NOT NULL,
    item_id        UUID NOT NULL,
    event_type     VARCHAR(16) NOT NULL,
    dwell_ms       INT,
    ts             TIMESTAMPTZ NOT NULL,
    INDEX idx_user_ts (user_id, ts)
) PARTITION BY RANGE (ts);

Hot serving state (not SQL):
  online_features:{user_id}   → recent-interaction feature vector, TTL minutes
                                 (what makes "clicked 10 min ago" usable NOW)
  ann_index:items             → HNSW/IVF index over item embeddings, served in-memory

The split mirrors Pastebin's metadata/blob split for a different reason: here it's not object size, it's update cadence. user_features/item_features are batch-refreshed (hours); the online feature store is refreshed per-event (seconds). Mixing them into one store means either the whole thing is slow-to-update (kills freshness) or the whole thing is written on every event (kills the batch training pipeline's ability to read a stable snapshot).

8. Version 1 — simplest thing that works¶

No personalization, no real-time anything. A batch job computes "most popular items" (or basic item-item co-occurrence: "users who interacted with A also interacted with B") once a day, and every user gets served from that precomputed list.

graph LR
    Batch[Daily batch job\ncount interactions] -->|write| Cache[(Precomputed list\nglobal or per-cohort)]
    Client -->|GET /recommendations| API[API server]
    API -->|lookup| Cache

# V1: daily batch job, no per-request personalization
def compute_popular_items(interaction_log_yesterday):
    counts = Counter()
    for event in interaction_log_yesterday:
        if event.type in ("click", "watch", "purchase"):
            counts[event.item_id] += 1
    top_items = counts.most_common(500)
    cache.set("popular:global", top_items, ttl=86400)

def get_recommendations(user_id, limit=20):
    # everyone gets the same list — or a coarse cohort (e.g. by region/category)
    return cache.get(f"popular:{cohort_of(user_id)}")[:limit]

This ships fast, is trivial to reason about, and beats an empty screen. Do not add ML infrastructure yet — confirm this is actually where it breaks.

9. Identify the bottleneck¶

10. Version 2 — embeddings + lightweight ranking, near-real-time features¶

Two additions: (1) represent users and items as embedding vectors and use approximate nearest-neighbor (ANN) search to generate candidates specific to this user, and (2) a lightweight model scores those candidates, reading features that update from recent interactions rather than only yesterday's batch.

Candidate generation via embeddings, conceptually: a model (trained offline on interaction history — e.g. two-tower: one tower encodes users, one encodes items, trained so that a user's vector is close to vectors of items they engaged with) produces a fixed-size vector per user and per item. "Find items this user would like" becomes "find item vectors close to this user's vector" — a nearest-neighbor search. Doing that exactly over 50M items per request is too slow, so an approximate index (e.g. HNSW — a layered graph structure where search hops through progressively finer layers to reach a near-neighborhood in roughly logarithmic time instead of scanning all 50M vectors) trades a small amount of recall for orders-of-magnitude faster lookup. You don't need to derive HNSW's internals in an interview — naming that ANN indexing exists and why (sublinear search over a high-cardinality vector space) is the expected depth.

graph TD
    Client -->|GET /recommendations| API[API server]
    API -->|1. fetch user embedding + recent features| Feat[(Online feature store)]
    API -->|2. ANN search| ANN[(ANN index over item embeddings)]
    ANN -->|~300 candidates| API
    API -->|3. score candidates| Rank[Lightweight ranking model]
    Rank -->|ranked top 20| API
    API --> Client
    Client -->|POST /events| EventQ[Event stream]
    EventQ -->|update| Feat

def get_recommendations(user_id, limit=20):
    user_vec = feature_store.get_user_embedding(user_id)
    recent = feature_store.get_online_features(user_id)   # last N interactions, minutes-fresh

    candidates = ann_index.search(user_vec, k=300)          # cheap: vector search
    candidates = filter_seen_and_ineligible(candidates, user_id)

    scored = ranking_model.score_batch(user_id, recent, candidates)  # one batched call
    return sorted(scored, key=lambda x: -x.score)[:limit]

def on_interaction_event(event):
    # near-real-time: feeds the NEXT request's candidate generation and ranking
    feature_store.append_online_feature(event.user_id, event.item_id, event.type, event.ts)

The key structural change from V1: feature_store.get_online_features reads state that was written minutes ago by the event stream, not by last night's batch job — this is what makes "clicked 10 minutes ago" actually show up in the next request.

11. Identify the next bottleneck¶

12. Version 3 — production architecture¶

Add multiple candidate sources (so no single failure or cold-start case yields an empty list), split the feature store into online (low-latency, event-driven) and offline (training-time, batch) halves, and make the ranking service a proper batched model-serving component.

graph TD
    Client -->|GET /recommendations| Gateway[API Gateway]
    Gateway --> CandSvc[Candidate Generation Service]

    subgraph "Candidate sources — run in parallel, merged"
      CandSvc -->|ANN over embeddings| ANN[(ANN index: HNSW)]
      CandSvc -->|fallback, always available| Pop[(Popularity index)]
      CandSvc -->|session context| Recent[(Recently-viewed-similar)]
    end

    CandSvc -->|~300-500 merged, deduped candidates| RankSvc[Ranking Service\nbatched model serving]
    RankSvc -->|features| OnlineFS[(Online Feature Store\nRedis, minutes-fresh)]
    RankSvc -->|ranked list| Gateway --> Client

    Client -->|POST /events| Stream[Event Stream\nKafka]
    Stream -->|near-real-time update| OnlineFS
    Stream -->|append| Log[(Interaction Log\ndurable, partitioned)]

    Log -->|nightly/hourly| Train[Offline Training Pipeline]
    Train -->|new embeddings, new ranking model| OfflineFS[(Offline Feature Store /\nEmbedding Store)]
    OfflineFS -->|sync| ANN
    OfflineFS -->|sync| OnlineFS
    Train -->|deploy| RankSvc

Key production decisions:

• Candidate generation fans out to multiple sources in parallel, not just ANN. Popularity is the cheap, always-available fallback (never zero results, matches V1's role exactly — it doesn't disappear, it becomes one input among several). Recently-viewed-similar handles the contextual "related items" surface using session signal rather than the standing user embedding.
• Ranking service is a proper batched model server — candidates for a request are scored as one batch (see Model Serving for continuous batching and why per-item scoring calls don't scale). Precompute/cache item-side features per candidate so the ranking call only needs to combine them with the user's features at request time, not recompute both from scratch.
• Online vs. offline feature store split, matching the freshness split from Section 4: online store (Redis-class, TTL'd, event-driven) answers "what has this user done in the last few minutes" for ranking; offline store (data warehouse / offline embedding store) is what the training pipeline reads to produce the next embedding and ranking model versions. They're kept in sync by periodic export, not by being the same store — training needs a stable snapshot, serving needs sub-second writes.
• Event stream (Kafka-class) decouples "user acted" from "feature updated." This is what makes near-real-time freshness possible without every event handler being on the request's critical path.
• Offline training pipeline runs on a schedule (hourly embedding refresh, daily/weekly full model retrain), reading the durable interaction log — the same log doubles as the audit trail and the labeled-training-data source (impression + subsequent click within a session = a positive label).

13. Failure analysis¶

14. Consistency considerations¶

• Most of this pipeline is eventual consistency by design, and that's fine. Embedding updates from the offline training pipeline propagate to the ANN index and feature stores on the order of hours — nobody notices a few hours' lag in "this item's learned embedding got slightly better."
• Near-real-time freshness (Section 4) is a bounded staleness requirement, not strong consistency. "Within minutes" is an SLA on an eventually-consistent pipeline (event stream → online feature store), not a demand for synchronous read-after-write.
• Where staleness visibly degrades UX: the online feature store lagging past its freshness budget (users notice "it doesn't know what I just did"), and the popularity fallback being stale during a fast-moving trending event (a viral item not yet reflected in the popularity index looks like a miss).
• Where it doesn't matter at all: offline feature/embedding freshness, training pipeline cadence, and the interaction log's replication lag to the data warehouse (nothing serving-critical reads from there directly).

15. Observability¶

Latency / infra metrics:
  recs_request_latency_p50/p99{stage=candidate_gen|ranking|total}
  ann_search_latency_p99
  ranking_batch_size, ranking_queue_wait_ms   (see Model Serving for the batching angle)
  online_feature_store_staleness_s
  event_stream_consumer_lag_s

ML-specific metrics — first-class, not an afterthought:
  recs_ctr{surface=home|item_detail}            click-through rate on shown items
  recs_engagement_rate                          watch/purchase rate among clicks
  recs_diversity_score                          category spread within a returned list
  cold_start_candidate_share                    % of served list from fallback vs. embedding path
  model_offline_vs_online_auc_delta             training-serving skew signal
  new_item_time_to_first_impression             validates the "minutes" freshness requirement

Alerts:
  recs_ctr drops > X% week-over-week for a cohort (silent relevance regression)
  ann_search_latency_p99 > budget
  event_stream_consumer_lag_s > freshness SLA
  cold_start_candidate_share spikes (embedding pipeline broken, not actually more new users)

16. Cost analysis¶

Ranking model serving (GPU/CPU, batched): the dominant line item — driven by
  candidates/request × requests/sec (Section 5's ~28M scoring-calls/sec peak).
  See Model Serving's cost model (GPU-hours, batch size, quantization) for the levers.

ANN index (in-memory, HNSW-class):        tens of GB (Section 5), sized to fit in memory
  across a small cluster                                          ~$1,500-3,000/mo

Online feature store (Redis-class):        ~100GB+ hot working set, replicated  ~$800-1,500/mo

Event stream (Kafka-class):                throughput-sized to interaction volume ~$500-1,000/mo

Offline training pipeline (batch compute, periodic not continuous): spiky,
  amortized well below serving cost                                ~$1,000-2,000/mo

Interaction log storage (object/columnar, cheap, append-only):      ~$200-500/mo

Cost lever: ranking model batch size and candidate-set size (300 vs 500 candidates
  per request) trade recommendation quality against the dominant serving cost line —
  this is the single biggest knob, same as the batch-size lever in Model Serving.

17. Alternative architectures¶

18. Staff Engineer Extensions¶

19. Interview follow-ups¶

1. Why can't you just rank the whole catalog for every user? Say the multiplication out loud: users × catalog size × per-item scoring cost is intractable at any real scale (Section 1's numbers) — this is the entire reason the funnel exists, and naming it precisely is worth more than any diagram.
2. How do you handle a brand-new item with zero interactions? Content-based candidate generation (seed an embedding from metadata/content) plus deliberate exploration slots in the candidate set so it can accumulate interaction signal — pure collaborative filtering structurally cannot surface it (Section 13).
3. How would you A/B test a new ranking model safely? Shadow scoring first to compare against logged outcomes without affecting users, then a staged rollout gated on engagement metrics specifically, not just latency/error rate (Section 18) — a broken ranking model is often invisible to infra monitoring.
4. What's different about serving a ranking model here versus a general LLM-serving problem? Candidate count per request is small and bounded (hundreds, not an open-ended generation), so the Model Serving batching mechanics still apply, but the interesting system design here is upstream of the model server — which candidates even reach it, and how fresh their features are — not the serving stack itself.

Self-Assessment¶

• [ ] I can explain the candidate-generation → ranking funnel and why it's not optional at scale, with the multiplication that proves it
• [ ] I can distinguish the two freshness requirements (new item vs. new user action) and name the mechanism that satisfies each
• [ ] I can describe cold-start handling for both new users and new items, and why they need different mitigations
• [ ] I can explain why click-through/engagement rate belongs on the observability dashboard alongside latency, and what a latency-only dashboard would miss
• [ ] I can describe shadow scoring and staged rollout for a new ranking model, and why a hard cutover is the wrong move