Multi-Region Architecture & Disaster Recovery¶

Prerequisites: Replication, Consistency Models

← Distributed Systems | Previous: Raft

Why This Exists¶

A single-region system has one region-sized SPOF (see Single Points of Failure, applied at the largest possible scale): the AWS us-east-1 outage, the datacenter that loses power, the fire suppression system that floods the server floor. None of these are exotic — they've all happened. The question a DR strategy answers isn't "can this happen" — it's "when it happens, how much data do we lose, and how long are we down?"

Those two numbers have names:

• RTO (Recovery Time Objective) — how long can the system be down before recovery completes?
• RPO (Recovery Point Objective) — how much data (measured in time) can be lost — i.e., how stale is the most recent backup/replica you fail over to?

Both can be driven toward zero. Neither is free — and the cost curve isn't linear, it's closer to exponential as RTO/RPO approach zero. The senior skill here is picking a point on that curve deliberately, tied to an actual business requirement, instead of reflexively saying "multi-region active-active" because it sounds impressive.

Mental Model: Two Dials, One Cost Curve¶

Cost / complexity
        │                                                    ● Active-Active
        │                                          ● Warm Standby
        │                                ● Pilot Light
        │                    ● Backup & Restore
        └────────────────────────────────────────────────────────
          RTO: days          hours        minutes        seconds
          RPO: hours          minutes      seconds        ~0

Each tier to the right buys you a smaller RTO/RPO by keeping more infrastructure warm, more data replicated, in more places, all the time — which is exactly what costs money and adds operational surface area. There is no tier that's simply "better" than another; there's only the tier that matches what the business actually needs, which is why the first question in any DR design is "what's the RTO/RPO requirement," not "what pattern should we use."

The Four DR Tiers¶

flowchart TB
    subgraph T1["1. Backup & Restore — RTO: hours-days, RPO: hours"]
        A1["Primary region"] -.->|periodic backup| S1[("Cold storage")]
    end
    subgraph T2["2. Pilot Light — RTO: ~1 hour, RPO: minutes"]
        A2["Primary region<br/>full capacity"] ==>|continuous replication| S2[("DR region<br/>DB running, compute OFF")]
    end
    subgraph T3["3. Warm Standby — RTO: minutes, RPO: seconds"]
        A3["Primary region<br/>full capacity"] ==>|continuous replication| S3["DR region<br/>scaled-down but LIVE"]
    end
    subgraph T4["4. Active-Active — RTO: ~0, RPO: ~0"]
        A4["Region A<br/>serving traffic"] <==>|bidirectional sync| S4["Region B<br/>serving traffic"]
    end

• Backup & restore: back up data periodically, store it (ideally in a different region/provider than the primary), and rebuild everything from scratch on disaster. Cheapest, slowest — RTO is however long it takes to provision infrastructure and restore data, which for a large database is measured in hours, sometimes days.
• Pilot light: the smallest possible version of the system runs permanently in the DR site — usually just the database, kept in sync via continuous replication — while compute stays off until needed. On disaster, you scale up compute around the already-warm data. RTO drops because the slowest part (data) is already there.
• Warm standby: a scaled-down but fully functional copy runs continuously in the DR site, capable of handling some traffic already. Failover means scaling it up and redirecting traffic, not building it from nothing.
• Active-active: both (or more) regions run at full capacity, serving live traffic simultaneously, with bidirectional replication. There's no "failover" step for reads/writes to the healthy region — traffic just stops routing to the dead one. This is also the hardest tier to get right: bidirectional writes mean you've taken on the multi-leader conflict problem (see Replication and CRDTs if writes need to merge automatically).

Cloud ↔ Datacenter Scenarios¶

Multi-region DR is usually taught as cloud-region-to-cloud-region. In practice, a large fraction of real DR architectures are hybrid — one side is a cloud region, the other is an on-prem datacenter — and this combination has failure modes the pure-cloud version doesn't.

flowchart LR
    subgraph DC["On-prem Datacenter"]
        DCApp["App tier"] --> DCDB[("Primary DB")]
    end
    subgraph Link["Dedicated link"]
        VPN["Direct Connect / ExpressRoute<br/>+ VPN backup path"]
    end
    subgraph Cloud["Cloud region (DR site)"]
        CloudApp["App tier<br/>(scaled down or off)"]
        CloudDB[("Replica DB")]
    end
    DCDB ==>|async replication| VPN
    VPN ==> CloudDB
    CloudApp -.-> CloudDB

Why this pairing is common, and why it isn't symmetric:

• DC primary, cloud DR site — the most common shape. Standing up a second physical datacenter purely as a cold DR target is expensive and slow to provision; a cloud region gives you elastic, pay-for-what-you-use pilot-light or warm-standby capacity without a second lease and a second hardware refresh cycle. This is frequently the first cloud workload a DC-heavy org runs — DR, not the primary app.
• Cloud primary, DC DR site — less common, but real: regulated industries where data residency, a specific compliance certification, or a contractual requirement mandates that a recoverable copy exist on infrastructure the company physically controls, independent of any single cloud vendor. Also shows up as insurance against a full cloud-provider-account-level failure (billing dispute, account suspension, provider-wide outage) that a same-vendor multi-region setup doesn't protect against.
• Mid-migration (both are "primary" for a while) — during a lift-and-shift or replatform, the DC and the cloud region run in parallel, often with the DC as source of truth and the cloud side catching up via replication until cutover. This is DR infrastructure and migration infrastructure using the identical mechanism (continuous replication + a cutover runbook) for different purposes — worth naming explicitly when asked, since interviewers sometimes conflate "migration plan" and "DR plan" and the honest answer is "they share a replication pipeline, but the failback story is different."

What's different from cloud-to-cloud DR:

Failback Is Harder Than Failover¶

Failover gets the attention because it's the dramatic, time-pressured half. Failback — returning to the original primary once it's healthy again — is usually the part that gets skipped in the design and then discovered the hard way during the actual incident.

• Every write accepted by the DR site during the outage now has to reconcile with the original primary before it can resume serving — if the DR site took writes (warm standby or active-active), that's a second multi-leader conflict problem, not just "switch DNS back."
• Failback in the DC-as-DR-target direction is often manual and untested, because DR drills tend to test failover (the scary direction) and skip failback (the "we'll figure it out" direction) — which means the first real failback is happening under the additional pressure of "we've now been running degraded for however long the DR window lasted."
• The only credible claim is "we've tested failover and failback, on a schedule." "We have a replica" and "we have successfully failed over and back within our RTO in the last quarter" are different claims, and only the second one is evidence.

sequenceDiagram
    participant P as Primary (original site)
    participant D as DR site
    participant Ops as Operators

    Note over P,D: FAILOVER — fast, rehearsed, time-pressured
    P--xP: Outage
    Ops->>D: Redirect traffic to DR site
    D->>D: DR site starts accepting writes
    Note over D: RTO/RPO clock is running

    Note over P,D: Time passes — DR site is now the de facto source of truth
    D->>D: Accumulates writes for the duration of the outage

    Note over P,D: FAILBACK — slower, often manual, frequently untested
    P->>P: Primary recovers, but its data is now STALE
    Ops->>P: Reconcile — replay/merge DR's writes into primary
    Note over P,D: A second multi-leader conflict problem —<br/>not just "switch DNS back"
    Ops->>P: Cut traffic back once reconciled

Realistic Example¶

A payments company runs its primary stack in an on-prem datacenter (regulatory requirement: card data must stay on infrastructure they directly control) and a warm-standby DR site in a cloud region connected via a dedicated Direct Connect link with a VPN failover path. Database changes replicate asynchronously to the cloud region (RPO target: 30 seconds under normal load). Quarterly, they run a full failover drill: redirect a slice of production traffic to the cloud DR site, verify correctness, then fail back — and track actual RTO/RPO achieved against the target, not just "did the drill complete." Two things they learned from drills, not from design review: their DNS TTL was longer than their RTO target (fixed by lowering it and adding a traffic-manager health check instead of relying on DNS alone), and their failback runbook assumed the DC's connection pool would recover cleanly, which it didn't under real load — surfaced only because they actually failed back under load, not just failed over.

Trade-offs¶

Interview Questions¶

Key Takeaways¶

See also: Replication for the mechanics of keeping copies in sync, Single Points of Failure for the redundancy levers this page applies at region scale, CRDTs for merging concurrent writes in an active-active topology.

Previous: Raft