Capstone — An Evaluated, Authorized, Tool-Using Production AI Service

The service already runs, which is why its failures are dangerous. A request can hang forever, a mixed billing-and-shipping ticket is misrouted, retrieval returns no evidence, authorization approves everyone, and every request pays for the large model. Your task is to close those planted holes one gate at a time and produce evidence that each failure is now bounded.

This is the course case in executable form: the story belongs to your test results, traces, denial paths, and measurements. Keep the specification as the source of truth; no fictional incident can substitute for a passing failure injection.

Start from capstone-starter/ at the repo root — a runnable support-ticket triage service with a mock model (no API keys) and five planted holes. Do not scaffold a new framework. Close the holes using the five-gate checkpoints. Learner ticks: capstone-starter/PROGRESS.md.

cd capstone-starter
pip install -r requirements.txt
uvicorn app:app --reload          # GET /healthz  POST /v1/triage
pytest tests/test_api.py -v       # must pass
pytest tests/test_eval.py -v      # suite runs; planted miss is reported on purpose

The five gates are learned in sequence; the capstone proves you can hold all five at once, in a single service, under the same request path. It is domain-agnostic — unlike the specialization tracks, which go deep on one vertical (stock research, hybrid architectures, an editor plugin) over 90 days, the capstone is the generic "does the whole chain actually work together" checkpoint. Run it instead of a track, or alongside one — a track's day-90 demo can double as your capstone if it satisfies every row below.

For the oral defense format (architecture sketch, live happy path, failure demo, Q&A), see Assessment rubrics → Capstone oral. This page is the build spec; that section is the defense format.

flowchart TB
  CS[Core service] --> Svc(("One production<br/>AI service"))
  Ev[Evaluation] --> Svc
  Kn[Knowledge] --> Svc
  Ag[Agent] --> Svc
  Op[Operations] --> Svc
  Sec[Security] --> Svc

The six parts

The six parts are the build checklist. Done means Gates 1–5. The decision-path row is Gate 6, and Gate 6 is a stretch, outside done. Capstones are off the XP track: the Full Core badge is the 28 modules, and finishing this service awards no XP.

Part	You must ship	Gate(s) it proves	Modules to have completed
Core service	Provider abstraction, structured/schema-valid output, deadlines on every model call, retries, fail-closed validation	1	01, 02, 03, 13
Evaluation	A golden set, deterministic checks, quality metrics, and a CI gate that can fail a build on regression	2	04, 22
Knowledge	Retrieval with citations, a measured decision on whether retrieval is even needed, freshness handling, retrieval metrics (recall/precision/groundedness)	3	05, 07, 09
Agent	Tool use with an enforced permission boundary, loop/step/cost caps, persisted state, human approval on at least one write action, a verifier that is not the model	4	08, 10, 11, 12, 20, 21, 27
Operations	Traces tied to a request_id, a cost dashboard, p50/p95/p99 latency, a documented fallback path, one rehearsed incident scenario	5	13, 17, 22, 23
Security	Authorization enforced outside the model (not a prompt instruction), parameter validation on every tool call, resilience to a prompt-injection test case, least privilege on anything that writes	1, 4	02, 08, 21
Decision path (stretch)	A real-time decision where the model picks from known options instead of writing: timed against writing, checked for letter bias, with a rule for unsure answers	6 (optional)	03, 04, 06, 17

Where teams lose the most time, and how to avoid it

The failure demo is the graded part. Gate 4 and Gate 5 are the reason this capstone exists, and they are the work that gets deferred. Kill the dependency and inject the payload early, while there is still time to change the design — not the night before.

Build the evaluation before the knowledge layer. Without Gate 2's numbers, you cannot tell whether retrieval helped, and "the demo looked better" is not a measured decision. The Knowledge row explicitly asks whether retrieval is needed at all; that question only has an answer if quality was measured without it first.

Authorization is a code path, not a prompt paragraph. Any instruction in a system prompt can be argued with by a sufficiently determined ticket. If the check is not a function the model cannot reach, Gate 4 is still open.

The architecture diagram must match the code. It is checked against what you built, not what you intended. Draw it last, or redraw it when the design moves.

Definition of done

• Demo runs from a clean clone with documented setup
• At least one automated test suite and one model/behavior eval, both runnable in CI
• An architecture diagram in the README that matches the code, not an aspirational one
• Known failure modes are written down, with which ones are handled and which ones aren't
• A live failure demo works: kill a dependency, feed it a prompt-injection payload, or force an empty retrieval, and show the system degrade the way it's supposed to
• Every claim in the "Operations" and "Security" rows above is demonstrated live, not asserted in a slide

Operational checkpoints (entry, build, eval, failure injection, exit, artifact) for each gate: Capstone gates. Tick them in capstone-starter/PROGRESS.md.

What "done" is not

A capstone that only has a happy-path demo has not closed Gate 4 or Gate 5 — those gates exist specifically because happy-path demos are where most production AI systems stop and where most production AI incidents start. Budget real time for the failure demo; it is graded, not optional.

Compare with the instructor reference

After making your own predictions, inspect the completed five-gate service. It adds authenticated principals, bounded model calls, a release gate, policy evidence, persisted approval with an idempotent simulated ledger, request traces, load measurement, and rollback. The student starter keeps its planted failures.

The other capstone

A specification-gated coding runtime takes the same five gates onto a different system: a 4B model may propose a patch, and the runtime decides whether the patch is legal. The triage service remains the capstone for one request path. The command runtime is the capstone for a 4B coding loop. Gate 6 on the triage checkpoints is the typed decision. The command runtime is what you build when the artifact still has to be written. Divide, solve, and join is the third capstone: a task a 20B model fails in one prompt, finished by splitting the work and joining the parts under checks you wrote.