Module 26 — Orchestrators in Production

Time: 5–7 days · Depends on: 12, 19, 25 · Pairs with: 08 MCP, 13 · Next: Specialization tracks

---

Learning objectives

• Compare custom loops, LangGraph, CrewAI, and MCP hosts with explicit trade-offs (control, HITL, durability, lock-in)
• Attribute cost and latency per agent/step, not only a monthly invoice
• Export reasoning traces as structured spans (replayable), not hidden essays
• Choose a stack from failure modes you can name, then keep Module 20–25 controls either way

---

Why this matters (CS engineer)

Two teams ship “multi-agent.” Team A copies a CrewAI demo: five personas, unbounded debate, $9/ticket, no step ids. Team B copies LangGraph: a 40-node graph nobody can draw, checkpoints on, still no spend guard, still auto-merge. Team C writes a 200-line custom loop, then rediscovers durable HITL the hard way. None of them can answer “which worker spent the money on step 7?” Frameworks are not villains. Unmeasured topology is.

Module 11 said learn the loop before LangGraph. This module is the comparison you can put in a design doc, plus the production numbers that make the comparison falsifiable.

Intuition lock

Sticky picture: Orchestrators are engines. Custom = building a kart (full control, you own brakes). LangGraph = a rail yard (graphs, checkpoints, HITL). CrewAI = an improv troupe (roles first, isolation last). MCP = USB-C peripherals (tools/resources), not the engine. Cost attribution is the itemized receipt. Traces are the black box recorder.

Kill this idea: “Pick the hottest agent framework and reliability appears.” → Replace with: Rank control, HITL, durability, ecosystem, and lock-in for your failure modes; keep breakers, sandboxes, evals, and per-step $ regardless of brand.

---

Mental model

flowchart LR
  subgraph choice [Pick from requirements]
    HITL["Need durable HITL?<br/><i>typical failure if forced anyway: no merge gate</i>"]
    Graph["Need branching graphs?<br/><i>typical failure if forced anyway: graph soup</i>"]
    Roles["Need role theater?<br/><i>typical failure if forced anyway: unbounded debate</i>"]
    Port["Need portable tools?<br/><i>typical failure if forced anyway: untrusted server as root</i>"]
  end
  HITL -->|yes| LG[LangGraph or custom+Module 25]
  Graph --> LG
  Roles -->|short tasks only| Crew[CrewAI]
  Port --> MCP[MCP host + any engine]
  Engine[Chosen engine] --> Attr[CostAttribution]
  Engine --> Trace[TraceRecorder]
  Engine --> Rel[Modules 20–23 controls]

Invariant: MCP can sit next to any engine. It does not replace LangGraph or a custom loop. Routing models is still Module 10.

---

Core tutorial

1. Comparison matrix (teaching ranks; 1 = best)

from src.orchestrators import compare_orchestrators, tradeoff_score

rows = compare_orchestrators("custom loop", "LangGraph", "CrewAI", "MCP hosts")
# columns: control, ops_cost, hitl, durable, ecosystem, lock_in
score = tradeoff_score(
    next(r for r in rows if r["name"] == "LangGraph"),
    {"hitl": 2.0, "durable": 1.0},
)

	Custom loop	LangGraph	CrewAI	MCP hosts
Control / testability	Highest — you wrote the state machine	High if you keep graphs small	Lower — personas hide control flow	Medium — policy is host-side
HITL / checkpoints	You build Module 25	First-class interrupt + durable state	Weak unless you bolt it on	Host approval UI; not a job queue
Durability	JSONL/WAL you own	Built-in checkpointers	Process memory unless you add store	Not a job graph: 2026 MCP has no protocol session; app state is explicit handles
Ecosystem	None	Large (LangChain world)	Fast demos, role packs	Many servers; supply-chain risk
Lock-in	Lowest	Medium (graph + APIs)	Medium	Protocol is open; hosts differ
Typical failure	Reimplement persistence badly	Graph soup; debug the library	Unbounded debate; weak isolation	Untrusted server as root
Use when	Topology is small and tests matter first	Branching + HITL + resume are load-bearing	Short role pipelines with caps	Portable tools across IDE and product

These ranks are ordinal teaching scores, not public benchmarks. Re-score with your weights (tradeoff_score). If HITL+durability dominate, LangGraph (or custom + Module 25) wins even if CrewAI demos prettier.

Explainer

Custom vs framework is a build-vs-buy on the control plane. Buy LangGraph when you would otherwise spend a quarter on checkpoints and resume. Stay custom when the graph would have six nodes and you already have Agent + Coordinator. CrewAI is a role abstraction; you still need Module 12 charters, budgets, and message schemas or you bought a costume. MCP is not in the same category as LangGraph — it is how tools show up. A serious design uses engine + MCP + host policy.

---

2. Failure analysis (how each dies in prod)

Stack	Runaway loops	Tool hallucination	Partial exec	Cost explosion	Silent degradation
Custom	You forgot max_steps	Allowlist in your registry	Your WAL / commits	Your SpendGuard	Your Module 22 suite
LangGraph	Graph cycles without a step cap	Node still calls whatever you bound	Checkpoint mid-write	Recursion limits ≠ $	Tracing without golden composite
CrewAI	Agents “discuss” uncapped	Shared tools, wide by default	No merge gate	Role fan-out	Demo metrics
MCP host	Server retries	Server exposes extra tools	Tool half-applied on the server	Unbounded resources	Host still 200

Regardless of stack, keep: Module 20 detectors + breakers, Module 21 manifests/sandboxes, Module 22 trajectories, Module 23 pins. Frameworks do not waive those.

---

3. Cost attribution per agent/step

from src.orchestrators import CostAttribution, CostEvent

led = CostAttribution()
led.record(CostEvent(
    agent="researcher", step=0, model="mini",
    tokens_in=100, tokens_out=40, usd=0.01, latency_ms=80, tool="search",
))
led.record(CostEvent(
    agent="writer", step=0, model="strong",
    tokens_in=400, tokens_out=200, usd=0.09, latency_ms=400,
))
led.by_agent()
# researcher vs writer itemized; total_usd()

This is the itemized receipt from the intuition lock, not a metaphor: by_agent() is the line-by-line breakdown, total_usd() is the total at the bottom. Put agent, step, model, tool on every span — those are the columns of the receipt. Monthly invoices cannot tell you the critic loop is 70% of $ — this can. Pair with Module 10 cost_per_success and Module 22 mean_spend_usd.

---

4. Observability of “reasoning”

Do not log raw chain-of-thought to a shared dashboard if policy forbids it. Do log:

from src.orchestrators import TraceRecorder

tr = TraceRecorder()
tr.span("decide", "researcher", tool="search", signature_hash="…")
tr.span("observe", "researcher", error=None, latency_ms=80)
tr.span("final", "writer", prompt_digest="abc")
tr.export()

Same fields as Module 11 step logs and OpenTelemetry. Langfuse/Phoenix are UIs over this shape. If you cannot replay, you cannot eval.

Think about it

Question: Product insists on CrewAI because a blog showed a “research team.” You need durable HITL, isolated writes, and $ per step. What do you actually ship?

Reveal a strong answer

Ship an **engine that has HITL and isolation** (custom Module 25 or LangGraph interrupts) plus **MCP or in-process tools** behind Module 21 gates. If you still want Crew-style *roles*, implement them as Module 12 charters on that engine — names in a YAML file are cheap; unbounded personas on a framework that does not pause are not. Measure vs a single-agent baseline (Module 12) before you keep the crew.

---

5. Emerging standards vs products

Layer	Standard / product	Owns
Tool/resource protocol	MCP	Discovery, invocation, (some) auth sketches
Agent graph runtime	LangGraph, custom, others	State, edges, checkpoints
Role packs	CrewAI, AutoGen, …	Persona orchestration
Eval / trace UIs	Phoenix, Langfuse, OTel	After-the-fact truth

Do not let a vendor slide collapse these layers. Write them as four boxes on the architecture diagram.

---

Failure modes

Symptom	Cause	Fix
“We picked X so we’re production”	Framework as talisman	List controls 20–25 still missing
Bill unexplained	No CostEvent.agent	Attribute or do not scale
Can’t replay	Prose logs	Structured spans + prompt digest
MCP vs LangGraph argument	Category error	Protocol vs engine
Matrix treated as science	Teaching ranks	Re-weight on your SLOs

---

Lab

1. compare_orchestrators for all four; write three sentences: when you’d pick each.
2. Record two CostEvents; assert writer > researcher in by_agent()["usd"].
3. TraceRecorder export has agent on every span.
4. Take one real workflow (even stubbed): name engine + MCP yes/no + which Module 20–25 controls you kept.
5. Optional: read LangGraph HITL docs after Module 25 and map interrupt → hitl events.

poetry run pytest tests/test_orchestrators.py -v

---

Quizzes

Quiz · +25 XP

Why is “LangGraph vs MCP” a bad comparison?

They are the same product from one vendor MCP standardizes tools/resources/prompts; LangGraph runs graphs/state — they stack MCP always includes LangGraph Neither is used in production

Quiz · +25 XP

What is cost attribution for in a multi-agent system?

Hiding the invoice from finance Assigning tokens, USD, and latency to each agent and step so you can kill expensive topology Using only the cheapest model forever A LangGraph-only feature

---

Open source materials

Resource	Use it for
src/orchestrators.py + tests	Matrix, attribution, traces
LangGraph	Graphs, HITL, checkpointers — after Module 25
CrewAI	Role crews — with Module 12 skepticism
MCP	Portable tools
12-factor agents	Control-plane principles

---

Checkpoint

• You can explain custom vs LangGraph vs CrewAI vs MCP without mixing categories
• You have a written pick for one workflow with weights
• Per-step $ / latency exist in a stub ledger
• Traces are structured and replayable
• Modules 20–25 controls are listed as kept, not “framework will handle it”

Mark complete when you can defend an orchestrator choice with trade-offs and show a per-agent receipt.

Complete module · +120 XP

Exercise

• Catalog: EX-26 — Orchestrator pick
• Prove: A written pick (custom vs hosted) plus per-agent cost and traces that include agent.
• Test: pytest tests/test_orchestrators.py -v

Next: Specialization tracks — add hardening, security review, and eval harnesses to the 90-day projects.