Sagas¶

Prerequisites: Local transactions, Messaging patterns

← Architecture Patterns | Next: Microservices Communication →

Why This Exists¶

Checkout is four writes that look like one business transaction:

1. Create order (order service)
2. Reserve inventory (warehouse)
3. Charge the card (payments)
4. Create shipment (shipping)

Each write lives in a different database. You charge the card. Shipping's API returns 503. The customer was billed. Nothing will leave the warehouse.

A local transaction cannot span those databases. Two-phase commit (2PC) can, on paper — and then a coordinator crash, a lock held across a network partition, and a 2-second payment gateway turn your checkout into a distributed deadlock. A saga accepts that each step commits locally, and that undo is your job: release the stock, refund the charge, mark the order cancelled.

The interview problem is not "what is a saga." It is: shipping fails after payment — what exactly runs, in what order, and what is still true if the refund itself fails?

Local TX vs 2PC vs Saga¶

flowchart TB
    subgraph Local["One database"]
      L1[BEGIN] --> L2[order + payment rows]
      L2 --> L3[COMMIT or ROLLBACK]
    end
    subgraph XA["2PC"]
      X1[Prepare all RMs] --> X2{All yes?}
      X2 -->|yes| X3[COMMIT all]
      X2 -->|no| X4[ABORT all]
      X2 -->|coordinator dies| X5[Locks held]
    end
    subgraph Saga["Saga"]
      S1[T1 commit] --> S2[T2 commit]
      S2 --> S3[T3 fails]
      S3 --> S4[C2 then C1]
    end
    style X5 fill:#b71c1c,color:#fff
    style S4 fill:#e65100,color:#fff

You pick a saga when you cannot lock the world for the duration of payment + warehouse + carrier APIs.

Choreography vs Orchestration¶

Choreography — each service emits events; others react.

OrderCreated → Inventory reserves → StockReserved → Payments charge
  → PaymentCaptured → Shipping ships
  → ShippingFailed → Payments refund → Inventory release → OrderCancelled

• Pro: no central brain, services stay decoupled.
• Con: the workflow is invisible. Adding a step means hunting consumers. Cycles and dual-writes to the outbox become a graph you debug in logs.

Orchestration — one coordinator (workflow engine or CheckoutSaga worker) tells participants what to do, records state, and runs compensations in reverse.

Orchestrator:
  execute Order.create
  execute Inventory.reserve
  execute Payments.charge
  execute Shipping.ship   ← fails
  compensate Payments.refund
  compensate Inventory.release
  compensate Order.cancel

• Pro: one state machine you can test, timeout, and show on-call.
• Con: coordinator is a dependency (make it durable, not a singleton JVM).

Compensations Are Not Rollbacks¶

A compensation is a new local transaction that semantically undoes a prior one.

Rules that keep you employed:

1. Idempotent forwards and compensations (same saga id / idempotency key).
2. Retry with backoff until a terminal state — or park for ops.
3. Never assume compensation ran because you sent the message.
4. Design the visible intermediate states (PAID_UNSHIPPED) instead of pretending they cannot happen.

Interactive Simulation¶

Inject Fail ship — payment has already committed. Watch compensations walk backwards.

Try: Fail charge — inventory reservation must release, order cancel, no refund. Fail ship — refund and release. Those are different undo lists.

How It Works Internally¶

Durable orchestrator (the production version of the sim):

saga_instance(id, type, state, payload, version)
saga_log(saga_id, step, action, status, attempt)

stateDiagram-v2
    [*] --> ORDER
    ORDER --> RESERVE: commit
    RESERVE --> CHARGE: commit
    CHARGE --> SHIP: commit
    SHIP --> DONE: commit
    CHARGE --> UNDO_RESERVE: charge fail
    SHIP --> UNDO_CHARGE: ship fail
    UNDO_CHARGE --> UNDO_RESERVE: refunded
    UNDO_RESERVE --> UNDO_ORDER: released
    UNDO_ORDER --> ABORTED
    DONE --> [*]
    ABORTED --> [*]

Each action is a local TX in the participant plus an outbox (or a sync RPC with a timeout + retry). The orchestrator advances only after a durable ack. Crash mid-step: on restart, read the log, retry the in-flight action (idempotent), do not skip.

Realistic Example: Shipping Fails After Payment¶

t0  Order  #991 PENDING          local TX
t1  Reserve SKU-44 × 1           local TX
t2  Stripe capture $49           local TX (PSP side + our payment row)
t3  Carrier API 503              ship step fails
t4  Refund $49                   may take 1500ms, or "pending"
t5  Release SKU-44
t6  Order #991 CANCELLED

What the user may see at t3–t4: charged. Support must have a page for PAID_UNSHIPPED. If t4 fails (Stripe down), you do not silently release inventory and walk away — money is gone, stock is free, the next customer buys the last unit. Park the saga in COMPENSATION_STUCK and page payments.

Compare 2PC: Stripe will not join your XA transaction. That option was never real.

Failure Modes¶

Compensation fails¶

• Symptom: saga stuck; money or stock wrong
• Fix: retry + dead-letter + operator runbook; never "delete the row"

Lost update / dual write¶

• Symptom: charged twice, or reserved twice
• Fix: idempotency key = saga id + step; unique constraint on (order_id, step)

Choreography cycle¶

• Symptom: event ping-pong, infinite refunds
• Fix: version the events; terminal states; orchestration for money

Partial visibility¶

• Symptom: recommendation service reads "paid" order that later cancels
• Fix: downstreams handle cancelled; or wait for DONE before side effects

Production Debugging¶

Symptom: customer billed, no tracking number

1. Find saga_id from order_id
2. Read saga_log — which step last succeeded? ship or charge?
3. Carrier and PSP dashboards for that idempotency key
4. If ship never succeeded: is compensation running or stuck?
5. If refund pending: do not re-run charge; wait or manual refund
6. Metrics: saga_stuck{state}, compensation_fail, step_latency

Trade-offs¶

Interview Questions¶

Key Takeaways¶

Previous: Kubernetes | Next: Microservices Communication