Core Modules¶
Twenty-eight modules, five gates. Sidebar numbers are stable IDs, not reading order. Each gate is a working-system exit criterion: you don't advance because you finished reading, you advance because the previous gate's residual failure mode forced the next capability. Complete Setup first. A module's Depends on line is the real prerequisite — gates group modules by which production failure they close, not by topic family, so a module you'd expect to sit elsewhere (cost optimization, MCP) may be grouped by the failure it actually prevents rather than the technology it uses.
flowchart LR
G1["Gate 1<br/>Dependable model service"] -->|"schema-valid ≠ correct"| G2["Gate 2<br/>Measurable quality"]
G2 -->|"tested ≠ grounded"| G3["Gate 3<br/>External knowledge"]
G3 -->|"grounded ≠ safe to act"| G4["Gate 4<br/>Actions and agents"]
G4 -->|"works on a laptop ≠ survives production"| G5["Gate 5<br/>Operate it"]
The running case¶
One thread ties the five gates together: a fictional support-ticket triage service. Each gate begins with a failure the current system cannot handle, adds an engineering capability, and leaves a narrower limitation that motivates later work. The incidents are teaching scenarios, not vendor postmortems.
Modules remain self-contained because this curriculum supports several reading paths. Each case states the system state it needs; callbacks add continuity but never require you to remember characters or read every earlier module. Follow Depends on for prerequisites. Sidebar numbers are stable IDs, not reading order.
| Gate | Problem entering the gate | Capability at exit | Residual limitation |
|---|---|---|---|
| 1 — Dependable model service | Soft output contracts and hostile input | Structured output checks, policy/input boundaries, fail-closed handling | Schema-valid output can still be wrong |
| 2 — Measurable quality | Behavior regresses without a failing test | Golden-set thresholds gate changes | Passing known tests does not supply missing business knowledge |
| 3 — External knowledge | Policy is missing, stale, or poorly retrieved | Budgeted retrieval with grounding and citation checks | Grounded answers do not authorize actions |
| 4 — Actions and agents | Tools can loop or act without authority | Runtime authorization, budgets, verification, and persistence | A safe local workflow still has to survive production |
| 5 — Operate it | Traffic, drift, outages, and cost are opaque | Observable, versioned operation with rehearsed rollback | Monitoring and incident response remain ongoing work |
Each gate section below names this same failure again, then shows the modules that close it.
Gate 1 — Dependable Model Service¶
An LLM call is an unreliable, nondeterministic, variable-latency, variable-cost distributed dependency. Before anything else is worth building, output has to be trustworthy: structured, schema-valid, and resistant to hostile input. This gate covers the contract — deadlines, versioning, and fail-closed serving are covered in full once you reach Production in Gate 5; here you build the discipline that makes that hardening possible.
Exit criteria
- Prompts are versioned, reviewable config — not ad hoc strings baked into code
- Hostile or untrusted input is sanitized/redacted before it reaches the model
- Model output is schema-validated (Pydantic/JSON schema), not parsed by string-matching
- Invalid or malformed responses fail closed instead of silently passing through
| Module | Time |
|---|---|
| 01 — Prompt engineering | 2–3 days |
| 02 — Security & privacy | 1–2 days |
| 03 — Advanced prompting | 3–5 days |
Gate 2 — Measurable Quality¶
Output is trustworthy in principle now — but "trustworthy" is unmeasured, which means every future change is a guess. Deterministic software gets unit and integration tests; stochastic AI behavior gets an evaluation suite — same engineering discipline, different tool. This is the course's load-bearing idea: skip it and every later gate is built on vibes.
Exit criteria
- A golden set (100+ cases) with a pass/fail threshold exists and runs in CI
- A prompt or config change can be shown to regress the eval score before it ships
- LLM-as-judge, if used, is checked against human-labeled agreement — not trusted blindly
| Module | Time |
|---|---|
| 04 — Testing & evals | 2–3 days |
Revisited for multi-step agents in 22 — Evaluating agentic systems (Gate 5) — trajectories need process and outcome scoring, not just a final-answer check.
Gate 3 — External Knowledge¶
A well-tested model that only knows its training data is still wrong about anything specific to your business. The next failure is ignorance — confidently wrong, not visibly wrong. This gate is where you decide whether retrieval is even necessary, and prove it with numbers instead of assuming it.
Exit criteria
- Context budget is enforced — no silent truncation of what the model sees
- Retrieval necessity is measured: raw-query baseline evaluated before adding rewriting, hybrid search, or reranking
- Retrieval quality has numbers — recall/precision, groundedness, citation correctness — not "it looked right in the demo"
- A fine-tune-vs-RAG decision is written down, not assumed
| Module | Time |
|---|---|
| 05 — Context engineering | 5–7 days |
| 06 — Fine-tuning | 7–10 days |
| 07 — Tools & basic RAG | 5–7 days |
| 09 — Advanced RAG | 7–10 days |
The answer contract is part of Module 07: one ticket leaves through answer, clarify, abstain, or escalate, including when the policy chunk and the billing tool disagree. Corpus engineering is part of Module 09: parent and child chunks, content hashes, and clauses that cite cleanly and still disagree. Neither page adds a module.
Gate 4 — Actions and Agents¶
Grounded answers are not the same as safe actions. Once the system can call tools and act across multiple steps, the failure mode changes again: loss of control — loops, hallucinated tool calls, partial execution, runaway cost. Authorization and budget enforcement have to live outside the model, because the model is exactly the thing that's unreliable.
Exit criteria
- Every tool/MCP server call is authorized outside the model (an allowlist, not a prompt instruction)
- Agents have a step cap and a cost cap enforced in code, not requested in the system prompt
- Named failure modes (loop, hallucinated tool, partial execution, silent quality drop) each have a detector and a test
- Any tool that writes or executes runs under least privilege / sandboxing
- Stop, verify, and persist live outside the model (a harness, not a longer prompt)
| Module | Time |
|---|---|
| 08 — Model Context Protocol | 4–6 days |
| 10 — Cost optimization | 2–3 days |
| 11 — Single-agent workflows | 7–10 days |
| 12 — Multi-agent coordination | 10–14 days |
| 18 — Agent design patterns | 5–8 days |
| 19 — Orchestration patterns | 6–9 days |
| 20 — Agent reliability & failure modes | 4–6 days |
| 21 — Secure tool use & sandboxing | 5–7 days |
| 27 — Harness engineering | 4–6 days |
The graph lesson sits on Module 19: the diamond, the speedup ceiling, and the knowledge context graph that refuses an illegal edge before the call. It does not add a module. Module 10’s router also carries a thinking-budget row: extra thinking tokens are spent only when a golden-set delta still fits the latency and dollar caps.
Gate 5 — Operate It¶
Everything above works on a laptop with one user. Production means real traffic, providers that rate-limit and change silently, audits, and a bill someone has to explain. This gate is where telemetry replaces guessing: every incident should be debuggable from traces, not from "the bot was weird."
Exit criteria
- Every egress model call has a deadline; failures are logged with a shared
request_id - Prompt/model/config version is recorded per request and roll-backable
- Dashboards track latency (p50/p95/p99), cost/request, success rate, retry rate
- Prompt/config drift is detected by a system, not discovered by a user complaint
- A serving change is kept only when a controlled comparison still meets quality and latency targets
| Module | Time |
|---|---|
| 13 — Production-grade systems | 2–3 weeks alongside a project. Weekend path: GET /healthz plus one trace that carries a request_id |
| 16 — Integration patterns | 1–2 weeks |
| 14 — Compliance | 3–5 days |
| 15 — Domain-specific applications | 1–2 weeks |
| 17 — Small & local models | 5–7 days |
| 28 — Inference serving | 2–4 days |
| 22 — Evaluating agentic systems | 5–7 days |
| 23 — Prompt & config drift | 3–5 days |
| 24 — Local-first, cost-aware agents | 4–6 days |
| 25 — Durable orchestration | 7–10 days |
| 26 — Orchestrators in production | 5–7 days |
The eval flywheel is the Gate 5 loop that turns a request_id into a redacted, frozen, quarantined golden row. It is not a 29th module.
Ready to prove all five gates work together? There are three capstones. The triage service starts from capstone-starter/. The command runtime starts from capstone-command/: a 4B model may propose a patch, and the runtime decides whether it is legal. Divide, solve, and join starts from capstone-decompose/: a 20B model fails the task as one prompt, and can finish it by dividing the work, solving the parts, and joining them under checks you wrote.
Full skill-by-skill breakdown: Capability progression. Prefer a guided route instead of the full list? See Learning paths.