# Instructor guide

For anyone delivering this as a workshop, a self-paced course, or a
partner-enablement session.

---

## Who this is for

The labs assume a working software engineer who has written HTTP services and
read JSON. They do **not** assume ML background, prior LLM work, or familiarity
with Anthropic products. Where LLM-specific vocabulary shows up, it gets defined
at first use in [`00-concept-map.md`](https://triage.mlynn.dev/docs/concept-map).

**Python-first rooms.** The labs are TypeScript and every learner should work
through them in TypeScript — the teaching is in the API, not the language, and
splitting a room across two runtimes doubles the failure modes you have to
debug from the front. There is a parallel Python implementation of
`/v1/triage` and the scoreboard, and the right way to use it is as *reading*,
not as an alternative track: point Python developers at
[the deltas page](https://triage.mlynn.dev/docs/python-deltas) during a break, or at the end. It
is four differences, and one of them is a mistake worth showing anybody who
will ever port a schema.

Learners who arrive with prior LLM experience usually arrive with *stale* prior
experience. `budget_tokens`, assistant prefill, "respond only with JSON"
prompting, edge-runtime-for-streaming. Surface that early, in the pre-flight
below. Otherwise it surfaces on its own as a confusing bug in Lab 3.

---

## Formats

| Format | Duration | Content |
|---|---|---|
| Lightning talk | 45 min | Concept map + live demo of all four routes |
| Half day | 3.5 hrs | Lab 0, Labs 1–4, concept map, architecture walkthrough |
| Full day | 6.5 hrs | Lab 0, Labs 1–6 + assessment + extension time |
| Day 2 — production | 4 hrs 10 | Labs 7–9: model choice, the trust boundary, and shipping it |
| Capstone | 45 min | Lab 10: Ask Northwind. Bolt onto either day, or send home |
| Self-paced | ~4 hrs | Day 1 labs; solutions unlocked per-lab |

**Day 1 is the course, and Day 2 is optional.** Say this to the room at the
start, because otherwise a two-day agenda reads as a two-day commitment and
you lose the people who only had a day.

Labs 0–6 are the enablement asset: the four capabilities, one domain, about
four hours. Someone who does only Day 1 has learned the Claude API and has a
working service to show for it. Nothing in Day 1 is a setup for Day 2 that goes
unpaid if Day 2 never happens.

Day 2 is for people who are going to ship. It is the decisions that only arrive
once something is real — which model, what happens when the input is hostile,
what separates a demo from a service — and it assumes the Lab 0 scoreboard
exists, because every measurement in it is a comparison against that baseline.

Booking only one day is a legitimate choice, and the right one for most
audiences. Book Day 2 when the room has a project rather than a curiosity.

**Lab 10 is the capstone and it belongs to neither day.** It is the only lab
whose subject is already deployed, so it needs no key, no local service, and no
terminal — a learner can work it from a phone on the train home. That makes it
the cheapest thing on this list to assign and the easiest to cut for time. Run
it live when the room has energy left at the close of Day 2; send it home
otherwise. It is genuinely valuable either way, and the section below says
which beats survive being read alone.

**Lab 0 is not optional and it is not a warm-up.** It is twenty minutes and it
establishes the scoreboard every later lab re-runs. Cutting it to save time
means Labs 2 and 5 have learners changing prompts with no way to tell whether
the change helped, which is the exact habit this course exists to break.

**Half-day timing (3.5 hrs including two breaks):**

| Time | Segment |
|---|---|
| 0:00–0:15 | Framing: the four-capability map, tier selection |
| 0:15–0:35 | Lab 0 |
| 0:35–0:50 | Lab 1 |
| 0:50–1:25 | Lab 2 |
| 1:25–1:35 | Break |
| 1:35–2:15 | Lab 3 |
| 2:15–2:40 | Lab 4 |
| 2:40–2:50 | Break |
| 2:50–3:20 | Architecture walkthrough (`docs/architecture.md`), Q&A |
| 3:20–3:30 | Where to go next |

Running behind, cut Lab 4 Steps 5–6 and the Lab 3 extensions. Never cut Lab 2.
It is the conceptual spine, and Labs 3 and 6 both build on it. Never cut Lab 0
either — see above.

---

## Pre-flight (send 48 hours ahead)

Learners who arrive without a working key lose 20 minutes, and they take a
neighbor's attention with them. Send this checklist:

- [ ] Send them [`setup.md`](https://triage.mlynn.dev/docs/setup). It covers everything below plus the
      two-terminal workflow and a troubleshooting table.
- [ ] Node 20+ (`node -v`)
- [ ] `jq` and `curl`. Every lab pipes JSON through `jq`, and it is the
      prerequisite people most often lack.
- [ ] An Anthropic API key with billing enabled. A key on an org with no credit
      fails at the first call with a message that reads like an auth error.
- [ ] `git clone && npm install && npm run smoke` completes
- [ ] Budget: measured costs are **~$0.10** for `npm run smoke` and **~$0.20**
      per `npm run eval`. With repeated runs across Labs 5 and 6, budget
      **$2–4 per learner** for the full sequence.

Have 2–3 spare keys on hand. Someone's will not work.

---

## What learners get wrong, by lab

### Lab 0

- **Skipping the hand-labelling in Step 2.** Learners want to run the command.
  Make them write three labels on paper first and compare with a neighbour. The
  disagreement on `NW-T-1060` is the whole lab — it is a multi-intent message
  against a single-label schema, so the argument is about the *schema*, not the
  ticket, and that reframe is what makes Lab 2 land.
- **Reading the score as a grade.** 10/12 is not a B. Push on what a one-case
  move can and cannot tell you; the second quiz item does this, but say it out
  loud too.
- **Not committing the baseline.** If nobody runs `--save`, nothing downstream
  has anything to compare against and the scoreboard steps in Labs 1–6 print
  "no baseline yet" for the rest of the day.

### Lab 1

- **`response.content[0].text`.** Everyone tries it. Let them hit the type
  error. It teaches the union better than a slide does, and then you can explain
  that adaptive thinking is on by default on this model family, so index 0 is
  frequently a `thinking` block rather than the text they were reaching for.
- **Confusing `max_tokens` with a budget.** The model cannot see it. It is a
  guillotine.

### Lab 2

- **Believing constrained output means correct output.** Step 3 exists to break
  this. The API guarantees the *shape*. Nothing guarantees the *content*.
- **Treating `.describe()` as comments.** The Step 2 experiment is the
  highest-value 8 minutes in the course. Do not skip it. Do not let a learner
  skip it because "the answer is obvious."
- **`parsed_output` non-null assertion.** Learners copy the `!` out of the docs.
  Ask what happens in production when it comes back null.

### Lab 3

- **Blaming the model for tool selection.** Step 2 reframes it as a description
  bug. That reframe transfers to almost every "the model is dumb" report you
  will ever get.
- **Not seeing the usage trap.** Some learners find Q3 genuinely hard. The key
  insight is that history grows each turn, so later turns carry more input than
  earlier ones, and the last turn is the largest one, which means reporting only
  that turn under-reports by less than 1/N and still badly. Sum every turn.
- **Uncapped loops.** If someone pulls `max_iterations` to see what happens,
  make sure they are watching their spend.

### Lab 4

- **Forgetting `curl -N`.** They will conclude streaming is broken. Take it as a
  gift. It is the Step 4 production bug, arriving early and for free.
- **`EventSource` in the extension.** It is GET-only. Discovering that is the
  point, so don't pre-empt it.
- **Assuming a non-2xx will surface stream errors.** Q4 is the one that saves
  them a production incident.

### Lab 5

- **Not restoring after breaks.** Have them `git diff` before moving on. A
  learner who leaves Break C in place gets confusing results in Lab 6.
- **Assuming caching is always a win.** Q5 is a real derivation. Make them do
  the arithmetic. The break-even is closer than they expect.

### Lab 6

- **Writing easy gold cases.** The instinct is to write cases the system passes.
  Push back. A case that always passes measures nothing.
- **Trusting the judge immediately.** Step 5 is not optional. A judge you have
  not validated is a random number generator with good manners.
- **Assuming a failure means the model is wrong.** Tell this one as a story,
  because it happened while this repo was being built. The first run of
  `evals/dataset.jsonl` scored 58%. Five of the six failures were *label*
  errors. `requested_remedy` had been labelled with the remedy the customer
  *should* get, and the schema asks for what they *explicitly asked for*, so on
  "My attorney will be contacting you about the tent poles that collapsed," the
  gold label said `escalation` and the model said `none` and the model was right.
  Fixing the labels moved the score without touching a line of prompt or code.

  The lesson that transfers: **when an eval fails, check the label before you
  check the model.** In mature systems a real share of "failures" are mislabeled
  cases. A gold set is a spec and it can have bugs like any other spec. The
  surviving cases in `dataset.jsonl` carry `notes` saying which rule each one
  tests, so the next person can tell a real regression from a disagreement about
  labels.

- **Judge variance is a teaching gift. Use it.** Three `npm run eval` runs scored
  the tone judge at 3/4, then 1/4, then 2/4 on a four-case sample. A 50-point
  swing, mostly unrelated to anything that changed. If a learner stumbles into
  this, don't treat it as a bug to fix. It is the most persuasive argument for
  Lab 6 Q1 you will ever get handed for free.

  Then push further, because the same runs found something real. The judge
  flagged replies promising a refund "today," which handbook clause 2.3 forbids.
  Refunds take 5-7 business days. The drafter system prompt had restated section
  1's *tone* rules and left clause 2.3 sitting unrestated in the handbook text,
  and under pressure the model did not go find it, so the fix was to hoist the
  hard constraints up into the role instructions, which is the same remedy
  `solutions/lab-6.md` Q4 prescribes.

  So one noisy signal was both too unstable to gate on and the thing that found
  a genuine prompt defect. Both are true at once. Holding both is the actual
  skill.

  Then make the hardest point. **After fixing, we deliberately did not re-run to
  show a better number.** At n=4 with that spread, a favourable run proves
  nothing, and tuning a prompt until a noisy judge approves is textbook
  overfitting to your own metric. The claim the evidence supports is narrow and
  qualitative. The judge's rationale on `eval-03` now credits the reply for
  stating the 5-7 day timeline. Ask the room what sample size they would need to
  support the *rate* claim instead. That is Lab 6 Q7, and most learners have
  never been asked to power-analyze an LLM eval.

---

### Lab 7

- **Reaching for the cheap tier before reading the budget.** The instinct is
  overwhelming and the lab is built to frustrate it. Make someone say out loud
  that $137/month against $4,000 means cost is not the binding constraint here.
  The transferable form: optimize the constraint that binds.
- **Reading the accuracy column and stopping.** Push them to the calibration
  gap. If the room only takes one thing from Day 2, it should be that a
  confidence score you have not checked for separation cannot carry a control.
- **Expecting the cheap tier to fail randomly.** It does not. It fails the
  multi-rule cases, and it fails `eval-04` — the safety case — at 0.95
  confidence. That single cell is the most persuasive thing on the page; put it
  on the screen.
- **Variance, again.** The matrix moves by up to two cases per model per run.
  Someone will get a run where Sonnet beats Opus on a case and want to build a
  theory on it. Point them back at Lab 0 Q5.
- **Timing.** `eval:models` takes 90 seconds and costs ~$0.19 per learner. If
  the room is on a shared key, run it once on the projector and have them read
  the checked-in matrix at `/playground/models` instead.

### Lab 8

- **Expecting prompt injection to be the whole lesson.** It is the hook. The
  substance is that `within_agent_authority` was a model-judged boolean nobody
  checked. Get someone to say out loud that the model was deciding whether the
  model was allowed.
- **Assuming a green gate means the defence is complete.** When this lab was
  built, the red-team run produced nine failures and **eight of them were
  mis-specified assertions**, including one that was literally inverted and two
  that failed the model for classifying a site-bug report as `other`. That is
  Lab 0's "check the label before the model" arriving in a new costume, and the
  case notes in `data/injections.jsonl` record it deliberately. Show them.
- **Forgetting the benign controls.** Three of the fourteen cases must NOT be
  blocked. Ask the room how they would score a defence that returns 400 for
  every request; the answer (12/12 on an attacks-only corpus) lands hard.
- **Skipping Step 5.** Measuring what the hardening cost is the step that
  separates this from security theatre. Budget for it.
- **Step 6 is about the control, not the model.** The room has spent an hour
  learning that model self-reports are hypotheses. Step 6 turns that on the
  guardrails themselves: the first `verifyCitations` flagged four real handbook
  clauses as fabricated, because it assumed a citation must come from a tool
  result when the whole handbook is sitting in the cached prefix. Ask who would
  have caught that. The answer is nobody on the accuracy suite — it looked like
  the model behaving badly, and it took a red-team run to see. If Lab 0's
  lesson is "check the label before the model," this is "check the control
  before you believe what it says about the model."
- **The temptation to gate on `cited_without_search`.** Someone will propose it,
  and it is a reasonable proposal. Walk it forward out loud: the gate goes red
  on runs where nothing went wrong, the team learns to dismiss it, and the day
  it fires for real it gets dismissed too. False positives spend the credibility
  the control needs on the one occasion it matters.
- **Timing and cost.** `eval:redteam` is ~90 seconds and ~$0.40 per learner —
  the most expensive single command in the course. On a shared key, run it once
  on the projector. Step 6 adds ~10 minutes and costs one `/v1/resolve` call;
  it is the step to cut if Day 2 is running long, and it reads well cold.

### Demoing the live queue

Worth five minutes at the end of Lab 8, and it lands better than any slide
about human-in-the-loop.

1. Open the storefront's [support form](https://northwind.mlynn.dev/support)
   and submit something a human must handle. The reliable one is a casually
   worded injury report — *"the bottle lining flaked and my kid swallowed a
   bit, probably nothing"* — because it is the October 2025 incident from the
   scenario, typed live.
2. Watch the pipeline reach `persist` and return a ticket id. Point out that
   the routine ticket you submitted earlier produced no id at all: storage is
   a consequence of escalation, not of submission.
3. Open `/queue?token=…` and work the ticket. Claim it, resolve it.
4. Open `/ops`. The escalation panel at the top is the only figure on that
   dashboard read from the database; everything below it is invented history,
   and both are badged. Ask the room which of the two they would trust in a
   board deck, and why the badge exists.

Two things to have ready: the `QUEUE_TOKEN` value, and the fact that the stored
message is redacted — expand the message on a card and show the
`[card ending 1111]` if someone submitted digits.

### Lab 9

- **The batch result is the lab.** Everyone arrives certain that batch is half
  price. It cost 23% MORE on this workload, because a cache read is 90% off and
  the two discounts compete on the same tokens. Do not spoil it — let them run
  all three commands and read the table. The moment the room realises why is
  the best five minutes of Day 2.
- **Do not overclaim it either.** At 400,000 tickets the prefix stays hot for
  hours and the result probably flips back. Lab 9 Q3 exists so nobody leaves
  believing "batch is a trap." The lesson is *measure*, not *avoid*.
- **Concurrency confusion.** Several learners will expect concurrency to cut
  cost. It cut wall clock 91s → 60s and nudged cost UP, because in-flight
  requests raced the cache write. Parallelism buys the clock, never the price.
- **`pool.ts` before `AdaptiveGate`.** Step 4 now opens on the forty-line
  bounded pool, and it is worth the two minutes because the room's instinct is
  binary: serial or `Promise.all`. Name the third option and connect it to Lab
  3 — unbounded concurrency against a metered API is the same shape of bug as
  an uncapped agent loop, and both are fixed by choosing a ceiling on purpose.
  Most workloads never need the adaptive version, and a fixed limit you
  understand beats an adaptive one you do not.
- **The MCP step needs a client.** Have Claude Desktop or Claude Code
  pre-configured on the projector machine, or the step becomes reading a file.
- **Timing.** Step 1 alone is ~6 minutes of wall clock and about $0.55 per
  learner — by far the most expensive command in the course. On a shared key,
  run the three modes once on the projector and have the room read the
  [batch planner](https://triage.mlynn.dev/playground/batch)
  instead.

### Lab 10

- **Reading it as an app tour.** The assistant is deployed and pretty, and a
  room will happily spend fifteen minutes chatting with it. The lab is about
  three boundaries, and the fastest way to get there is Check-your-work step 3:
  have someone request a $900 refund on the projector and get an escalation
  back. Then ask *why* it escalated. The answer people give is "the model
  followed policy." The answer is that `underAuthority` rewrote the outcome —
  the same `enforceAuthority` move from Lab 8, arriving where the model was
  actually being persuasive rather than merely being attacked.
- **Missing that the agent has no write tool.** This is the beat, and it is
  easy to read past because nothing on the screen announces it. Say it as a
  question: *"which of its four tools files the ticket?"* None of them. The
  confirmation is the write, and it is a separate request that re-derives
  authority, because a stored proposal is not evidence that it was ever within
  policy.
- **Assuming the course assistant is restricted by prompt.** It is not told to
  avoid support actions; it is not given the tools. Withholding a capability is
  a stronger guarantee than instructing against it — and that contrast is worth
  drawing explicitly against Lab 8, where prompt hardening was the *last*
  resort and bought only a probability.
- **Second-order injection.** Everyone escapes the user's message by now. Few
  expect `get_current_context` — a page title the browser supplied — to need
  the same treatment. Instruction-shaped text in a tool result arrives wearing
  the authority of a system-provided fact. If one idea from this lab survives
  the year, make it this one.
- **The Agent SDK argument is the most portable thing here, and rooms want to
  argue with it.** Let them. The claim is narrow and it is not "the Agent SDK
  is heavy": it is that a dependency configured with `tools: []`,
  `settingSources: []` and memory off has been switched off, and what remained
  was `messages.toolRunner`. Ask what would have to change about this
  assistant to flip the answer. Anyone who says "a filesystem" or "a shell" or
  "memory across sessions" has understood it.
- **Timing and cost.** ~45 minutes, and **$0 per learner on the deployed
  sites** — the storefront pays, and it is rate-limited and spend-capped. The
  only lab you can assign to a room with no keys at all.

## Knowledge checks

Twenty-one checks sit inline in the labs — two per lab, except Lab 4 which has
one — at the point the idea is taught rather than
collected at the end. They are authored as ` ```quiz ` JSON blocks in the
markdown and rendered as components by a remark plugin, so the source stays
plain markdown and still reads on GitHub.

Two things about how they behave, both worth knowing before you point a room
at them. The explanation shows on every answer, right or wrong, because a
check that only explains failures teaches the people who guessed correctly
nothing. And there is no retry or score — these are checks on understanding,
not an exam, and the exam is a separate page.

Use them as pacing. If half the room is still reading when the rest have
answered, that is your signal to move.

## Discussion prompts that work

Use these when a group finishes early or the room goes quiet:

1. *"You have a working `/v1/triage`. Your PM asks for 99.9% accuracy. What do
   you tell them?"* Surfaces the gap between "the model is wrong" and "the labels
   are ambiguous," which is where most eval work actually lives.
2. *"Which of the four routes would you delete first if you had to ship
   tomorrow?"* Good tier-selection reasoning. The usual answer is `/resolve`, and
   the usual reason is that a workflow could replace it.
3. *"Your cache hit rate drops from 94% to 11% overnight. Nobody deployed. What
   happened?"* Traffic-pattern reasoning, TTL expiry, tenant fan-out.
4. *"What in this repo would you not ship?"* If nobody bites, the omissions list
   at the end of `docs/architecture.md` is a good backstop.

---

## Grading the assessment

[`assessment.md`](https://triage.mlynn.dev/docs/assessment) has 16 questions, and there is an auto-scored
version at [/assessment](https://triage.mlynn.dev/assessment)
that marks sections 1 and 2 and walks learners through self-assessing section
3. Suggested weighting:

| Section | Questions | Weight |
|---|---|---|
| Mechanics (API surface, parameters) | 1–7 | 30% |
| Diagnosis (given a symptom, find the cause) | 8–12 | 40% |
| Design judgment (open-ended) | 13–16 | 30% |

Diagnosis carries the most weight on purpose. Reciting that `cache_control`
exists is worth very little. Recognizing a flat-zero `cache_read_input_tokens`
as a timestamp in the prefix is the skill that transfers.

Questions 13–16 have no answer key, so score the reasoning. A learner who picks
the "wrong" option with a sound cost and latency argument should outscore one
who picks the "right" option from memory.

---

## Adapting this to another domain

The structure ports. Replace four things:

1. `data/policies.md` — the large stable document that gets cached
2. `src/schemas.ts` — the structured output contract
3. `src/tools/` — the systems the agent queries
4. `evals/dataset.jsonl` — the gold set

Everything else stays: usage accounting, error mapping, SSE, prompt assembly,
the lab scaffolding. Budget about a day for a port, and most of that day goes to
hand-labelling a gold set rather than to writing code, which is the right place
for the time to go.

**One warning.** The domain needs a *real* policy document with genuine edge
cases. A toy domain produces toy labs, because every interesting question in
Labs 2, 3, and 6 comes from a place where two rules touch.

Source: https://triage.mlynn.dev/docs/instructor-guide
