Pass threshold: average ≥ 3.0 on required criteria, with no required criterion at 0.
Study cycle (predict → run → compare → explain): before each lab, write what you expect to happen (or fail). Run the smallest artifact. Compare the outcome to that prediction. Explain the gap in PROGRESS.md. Mark a module complete only after that cycle — a first-pass quiz is not a pass.
Five levels, one per depth of understanding — use these to check whether a student actually owns a gate's material or can only recite its vocabulary. Each level up requires the one below it; a student who can Design but not Predict is pattern-matching, not reasoning from the mechanism.
Level
Tests
4 looks like
Explain
Why the mechanism exists at all
A causal explanation in the student's own words, not a definition copied from the module
Predict
Forward simulation — given a change, what breaks
Names the specific failure and the mechanism that produces it, not "it gets worse"
Diagnose
Reading a trace/log/symptom and naming the failure
Names the failure mode and the evidence in the trace that points to it — a correct guess without evidence doesn't count
Design
Proposing a concrete fix
A specific, testable mechanism (code, config, or policy) — restating the principle is not a design
Defend
Justifying a decision against a real objection
Names the tradeoff that was accepted and why, not a restatement of the original choice
Explain: Why is an LLM API call a different kind of dependency than a deterministic library call?
Predict: A provider's median latency doubles overnight. If no request has a deadline, what happens to worker utilization under sustained load?
Diagnose: A response passes JSON parsing but fails your Pydantic schema on a field type. What's the actual failure — and is "just retry" the right fix?
Design: Add a fail-closed path for schema-invalid output that doesn't silently drop the user's request.
Defend: Why does temperature 0 not remove the need for an eval suite (Gate 2)?
Explain: Why do deterministic code paths get unit tests while model behavior gets an eval suite instead?
Predict: A one-line prompt "polish" ships without touching the golden set. What's the earliest point this course's CI setup could have caught a regression — and would it have?
Diagnose: Golden-set accuracy holds steady but user complaints rise. What's not being measured?
Design: Add a CI gate that blocks a merge when eval score regresses past a threshold, without blocking on noise from run-to-run variance.
Defend: Why is LLM-as-judge only trustworthy after it's checked against human-labeled agreement — what fails if you skip that check?
Explain: Why must tool authorization live outside the model instead of in a system-prompt instruction?
Predict: An agent's max_steps cap is set, but duplicate-tool-call detection is not. Where does the budget actually get spent when the agent loops?
Diagnose: A trace shows the same tool called four times in a row with near-identical arguments. What failure mode is this, and what's the earliest step it could have been caught?
Design: Add loop-control and tool-safety mechanisms to a single-agent loop that currently only checks a step counter.
Defend: Why is a prompt-injection classifier a mitigation, not a security boundary — what does it not protect against that an allowlist does?
Explain: Why does "it works in the notebook" not imply "it's a production service"?
Predict: A provider starts rate-limiting during a traffic spike, and no request has a timeout. Trace the failure through workers, health checks, and the autoscaler.
Diagnose: Given an Agent Flight Recorder trace with rising retry_count and flat success_rate, what's developing, and is it visible on a dashboard that only tracks success rate?
Design: Add a dashboard alert that would have caught the Gate 5 incident story before the bill did.
Defend: Why must a rollback path exist for prompt/model/config versions specifically, separate from a code rollback?