Pass threshold: average ≥ 3.0 on required criteria, with no required criterion at 0.
Study cycle (predict → run → compare → explain): before each lab, write what you expect to happen (or fail). Run the smallest artifact. Compare the outcome to that prediction. Explain the gap in PROGRESS.md. Mark a module complete only after that cycle — a first-pass quiz is not a pass.
Five levels, one per depth of understanding — use these to check whether a student actually owns a gate's material or can only recite its vocabulary. Each level up requires the one below it; a student who can Design but not Predict is pattern-matching, not reasoning from the mechanism.
Level
Tests
4 looks like
Explain
Why the mechanism exists at all
A causal explanation in the student's own words, not a definition copied from the module
Predict
Forward simulation — given a change, what breaks
Names the specific failure and the mechanism that produces it, not "it gets worse"
Diagnose
Reading a trace/log/symptom and naming the failure
Names the failure mode and the evidence in the trace that points to it — a correct guess without evidence doesn't count
Design
Proposing a concrete fix
A specific, testable mechanism (code, config, or policy) — restating the principle is not a design
Defend
Justifying a decision against a real objection
Names the tradeoff that was accepted and why, not a restatement of the original choice
Explain: Why is an LLM API call a different kind of dependency than a deterministic library call?
Predict: A provider's median latency doubles overnight. If no request has a deadline, what happens to worker utilization under sustained load?
Diagnose: A response passes JSON parsing but fails your Pydantic schema on a field type. What's the actual failure — and is "just retry" the right fix?
Design: Add a fail-closed path for schema-invalid output that doesn't silently drop the user's request.
Defend: Why does temperature 0 not remove the need for an eval suite (Gate 2)?
Explain: Why do deterministic code paths get unit tests while model behavior gets an eval suite instead?
Predict: A one-line prompt "polish" ships without touching the golden set. What's the earliest point this course's CI setup could have caught a regression — and would it have?
Diagnose: Golden-set accuracy holds steady but user complaints rise. What's not being measured?
Design: Add a CI gate that blocks a merge when eval score regresses past a threshold, without blocking on noise from run-to-run variance.
Defend: Why is LLM-as-judge only trustworthy after it's checked against human-labeled agreement — what fails if you skip that check?
Explain: Why must tool authorization live outside the model instead of in a system-prompt instruction?
Predict: An agent's max_steps cap is set, but duplicate-tool-call detection is not. Where does the budget actually get spent when the agent loops?
Diagnose: A trace shows the same tool called four times in a row with near-identical arguments. What failure mode is this, and what's the earliest step it could have been caught?
Design: Add loop-control and tool-safety mechanisms to a single-agent loop that currently only checks a step counter.
Defend: Why is a prompt-injection classifier a mitigation, not a security boundary — what does it not protect against that an allowlist does?
Explain: Why does "it works in the notebook" not imply "it's a production service"?
Predict: A provider starts rate-limiting during a traffic spike, and no request has a timeout. Trace the failure through workers, health checks, and the autoscaler.
Diagnose: Given an Agent Flight Recorder trace with rising retry_count and flat success_rate, what's developing, and is it visible on a dashboard that only tracks success rate?
Design: Add a dashboard alert that would have caught the Gate 5 incident story before the bill did.
Defend: Why must a rollback path exist for prompt/model/config versions specifically, separate from a code rollback?
The same defense fits the triage service, the command runtime, or divide, solve, and join. For the runtime, the failure demo is a free-form sentence that must not produce a patch, or a diff that leaves the allowlist. For the divide-solve-join capstone, the failure demo is a one-shot 20B result that must not count as done, or two green parts whose join fails the check you wrote.
Architecture sketch (2 min)
Live happy path (5 min)
Failure demo (a denied action, an empty retrieval, a sentence that must not patch, a diff outside the allowlist, or a join that fails the check you wrote) (3 min)
What you would build next (2 min)
Q&A (3 min)
Pass: Happy path works; one failure handled; student can explain tradeoffs.