Design: Pastebin¶

Difficulty: Foundation | Time: 45–60 minutes

1. Problem Statement¶

Design a service like Pastebin: a user pastes text, gets back a short URL, and anyone with that URL can read the text back. Optional: expiration, syntax highlighting, private pastes, burn-after-read.

This looks like the URL shortener. It is not. A shortened URL points away from you; a paste's content is your storage problem. Do not reuse the shortener's architecture without checking which assumptions still hold.

2. Clarifying Questions¶

3. Functional Requirements¶

• Create a paste (text body, optional title, optional TTL) → get a short URL
• Retrieve a paste by its short URL
• Optional: custom expiration, burn-after-read, syntax highlighting hint, private (auth-gated) pastes

4. Non-Functional Requirements¶

5. Capacity Estimation¶

Writes:
  1M pastes/day → ~12 writes/second average, ~120/s at 10× peak

Storage:
  Avg paste: 10 KB, p99: 1 MB, max: 10 MB (assume a hard cap)
  1M pastes/day × 10 KB avg ≈ 10 GB/day ≈ 3.65 TB/year
  This does NOT fit "cache everything" the way 30 GB of URLs did.

Reads:
  10:1 ratio → ~10M reads/day ≈ 115 reads/second average, ~1,150/s peak
  Mostly cold (read-once) — cache hit rate will be LOWER than a URL shortener's.

Bandwidth:
  Write: 120/s × 10 KB avg ≈ 1.2 MB/s (bursty up to 120 x 10MB if abused)
  Read:  1,150/s × 10 KB avg ≈ 11.5 MB/s

6. API Design¶

POST /api/v1/pastes
Request:  { "content": "...", "title": "optional", "ttl_seconds": 86400, "visibility": "unlisted" }
Response: { "paste_id": "a1B2c3D", "url": "https://paste.io/a1B2c3D", "expires_at": "..." }
Status: 201 Created, 413 Payload Too Large if over the size cap

GET /{paste_id}
Response: raw text or rendered HTML page
Status: 200, 404 (not found or expired), 410 Gone (burned)

DELETE /api/v1/pastes/{paste_id}   -- creator only, requires ownership token or auth

7. Data Model — split metadata from content¶

The URL shortener kept everything in one row because a URL is ~200 bytes. A paste can be 10 MB. Putting that in a relational row bloats indexes, wrecks buffer-pool cache efficiency, and makes every backup/replication cycle drag megabytes of blob through a system built for rows.

-- Metadata: small, hot, indexed. Lives in a fast KV/relational store.
CREATE TABLE pastes (
    id            VARCHAR(12) PRIMARY KEY,
    owner_token   VARCHAR(64),          -- hashed
    title         VARCHAR(200),
    size_bytes    INT NOT NULL,
    blob_key      VARCHAR(128) NOT NULL, -- pointer into object storage
    visibility    VARCHAR(16) DEFAULT 'unlisted',
    burn_after_read BOOLEAN DEFAULT FALSE,
    created_at    TIMESTAMP DEFAULT NOW(),
    expires_at    TIMESTAMP,
    INDEX idx_expires (expires_at)
);

Content: large, write-once, read-rarely-hot. Lives in object storage (S3/GCS), keyed by blob_key.
  s3://pastes-bucket/{shard}/{blob_key}

8. Version 1 — simplest thing that works¶

Single API process, Postgres for metadata, S3 for content. No cache yet.

graph LR
    Client -->|POST/GET| API[API process]
    API -->|metadata| PG[(Postgres)]
    API -->|blob get/put| S3[(Object Storage)]

def create_paste(content: bytes, ttl_s: int) -> str:
    paste_id = generate_id()  # base62, 7 chars, same scheme as URL shortener
    blob_key = f"{paste_id[:2]}/{paste_id}"   # prefix shard avoids hot S3 partitions
    s3.put_object(Bucket="pastes", Key=blob_key, Body=content)
    db.execute(
        "INSERT INTO pastes (id, blob_key, size_bytes, expires_at) VALUES (%s,%s,%s,%s)",
        paste_id, blob_key, len(content), now() + ttl_s
    )
    return paste_id

def read_paste(paste_id: str) -> bytes:
    row = db.query_one("SELECT blob_key, expires_at FROM pastes WHERE id=%s", paste_id)
    if not row or row.expires_at < now():
        raise NotFound()
    return s3.get_object(Bucket="pastes", Key=row.blob_key)["Body"].read()

This handles the average case fine. Do not add a cache yet — find the actual bottleneck first.

9. Identify the bottleneck¶

10. Version 2 — cache metadata AND content, but sized correctly¶

Two caches, deliberately different from each other, because metadata and content have different sizes and different hit-rate curves.

graph LR
    Client --> API[API pods]
    API -->|1: metadata lookup| MC[(Redis: metadata cache\nsmall, high hit rate)]
    MC -->|miss| PG[(Postgres)]
    API -->|2: content, only if small| CC[(Redis: content cache\nsize-capped, e.g. <= 100KB)]
    CC -->|miss / too large| S3[(Object Storage)]

def read_paste(paste_id: str) -> bytes:
    meta = metadata_cache.get(paste_id) or load_and_cache_metadata(paste_id)
    if meta.expired:
        raise NotFound()
    if meta.size_bytes <= CONTENT_CACHE_MAX:          # e.g. 100 KB
        cached = content_cache.get(meta.blob_key)
        if cached:
            return cached
        blob = s3.get_object(Bucket="pastes", Key=meta.blob_key)["Body"].read()
        content_cache.set(meta.blob_key, blob, ttl=300)
        return blob
    return s3.get_object(Bucket="pastes", Key=meta.blob_key)["Body"].read()  # large: always S3, use range/streaming

11. Identify the next bottleneck¶

12. Version 3 — production system¶

graph TD
    Client --> CDN[CDN edge cache\npublic/unlisted, NOT burn-after-read or private]
    CDN -->|miss| LB[Load Balancer]
    LB --> API[API pods]
    API -->|rate limit check| RL[Rate Limiter]
    API -->|metadata| MC[(Redis metadata cache)]
    MC -->|miss| PGr[(Postgres read replica)]
    API -->|content, small| CC[(Redis content cache)]
    CC -->|miss| S3US[(S3 us-east)]
    API -->|write path| PGw[(Postgres primary)]
    PGw --> PGr
    API -->|async scan| Scan[Content scanning queue]
    S3US -.->|cross-region replication| S3EU[(S3 eu-west)]
    TTL[TTL sweeper\nindex-driven, batched] --> PGw

Key production decisions:

• CDN in front of cacheable reads only. Public/unlisted pastes that are not burn-after-read are immutable once created — aggressive CDN caching (Cache-Control: immutable, keyed by paste_id). Burn-after-read and private pastes must not be CDN-cached — Cache-Control: no-store and origin-only. A cached burn-after-read paste is served after it has already been "consumed"; a cached private paste leaks past the auth gate. This is the global-latency lever for the public/unlisted majority, not a blanket in-front-of-everything box.
• TTL cleanup via index, not table scan. idx_expires (expires_at) lets a sweeper batch-delete WHERE expires_at < now() LIMIT 1000 cheaply, repeated. Never a full scan.
• Rate limiting at the edge (see the rate limiter exercise for the mechanism) — cap creates per IP/account before they hit Postgres or S3 at all.
• Async content scanning. Don't block the write path on malware/abuse scanning. Accept, store, return the URL, scan asynchronously, and yank (visibility = 'removed') if flagged. Blocking writes on a scanning service turns their outage into yours.
• Burn-after-read via atomic conditional delete. DELETE FROM pastes WHERE id=%s AND burn_after_read=true RETURNING blob_key — the row is gone after the first successful delete; a second concurrent read finds nothing and 404s. No separate lock needed.

13. Failure analysis¶

14. Consistency considerations¶

• Read-your-writes required: the creator must see their own paste immediately — write to primary, and either read from primary for a short window post-create or return the content directly in the create response so the client doesn't need to re-fetch at all.
• Eventual consistency acceptable for everyone else: a few hundred ms before a paste is globally cacheable/replicated is fine.
• Burn-after-read must be strongly consistent: this is the one place "eventually" is wrong — use the atomic conditional delete shown above, not a read-then-delete two-step (which races).

15. Observability¶

Key metrics:
- read_latency_p50/p95/p99 (SLO: p99 < 100ms)
- write_latency_p99
- cache_hit_rate{cache=metadata|content}
- s3_get_latency_p99
- ttl_sweeper_lag_seconds
- scan_queue_depth
- paste_size_distribution (catches abuse: sudden shift toward max-size pastes)

Alerts:
- p99 read latency > 250ms
- cache hit rate (metadata, hot public/unlisted only) < 90%
  (do not alert on a global 90% — most pastes are read-once and will miss)
- ttl_sweeper_lag > 15 minutes
- scan_queue_depth growing for > 10 minutes

16. Cost analysis¶

S3 storage (3.65 TB/year, growing):        ~$85/month (Standard) or less w/ lifecycle → IA/Glacier for expired-soon
S3 requests (10M reads + 1M writes/day):    ~$150/month
Postgres (metadata only, small rows):       ~$200/month primary + 1 replica
Redis (metadata + capped content cache):    ~$150/month
CDN (10M reads/day × 30 ≈ 300M requests/month): ~$1,500/month (at $0.005/1K requests, blended)
Total:                                      ~$2,085/month

17. Alternative architectures¶

18. Staff Engineer Extensions¶

19. Interview follow-ups¶

1. "How is this different from the URL shortener?" — Content size and read pattern. The shortener bets on "tiny object, cache everything, DB row is enough." Pastebin bets on "large object, split metadata from blob, cache selectively by size." Say this distinction explicitly — it's the whole exercise.
2. "How would you support editing a paste?" — Breaks the "immutable → CDN-cacheable" assumption. Either version pastes (new blob_key per edit, old URL redirects to latest or a specific version) or accept that editable pastes can't be edge-cached the same way.
3. "How do you prevent someone scripting mass uploads of stolen credential dumps?" — Rate limit creates per IP/account; async content scanning (hash against known-bad databases, entropy/pattern heuristics); fast takedown path once flagged; consider requiring auth above a certain paste-per-hour threshold.
4. "What if burn-after-read needs to work across regions?" — The atomic conditional delete must happen against a single authoritative row, not per-region replicas — route burn-after-read pastes' metadata writes/deletes to one home region even if content is replicated.

Self-Assessment¶

• [ ] Can I explain, with numbers, why this design splits metadata from content while the URL shortener doesn't?
• [ ] Can I justify capping the content cache by size instead of caching every blob?
• [ ] Can I walk through why burn-after-read needs an atomic conditional delete, not read-then-delete?
• [ ] Can I explain why a CDN matters more here than in the rate limiter or URL shortener exercises?
• [ ] Can I estimate storage cost growth over a year and name the lever (lifecycle policies) that controls it?