# Lab 7 answers

**Q1a. Do the two cost discoveries change the tier decision?**

No, and saying why is the whole exercise.

They do not change the decision, because the decision was never about cost.
Moving triage from Opus to Sonnet saves about $63 a month at warm rates ($111
versus $48). Fixing Haiku's cache would have taken it from ~$77 to ~$20. Against
a $4,000 budget, all of those round to free. An argument that was already
dominated by the accuracy column stays dominated by the accuracy column.

What they change is your confidence in every other number that table produced.
The matrix contained two cost errors in one column and nothing flagged either:
a fivefold cache miss on Haiku, and cold-write overhead that made Sonnet look
barely cheaper than Opus. Not a test, not a review, not the person who wrote the
"Haiku costs about half" sentence. The correct update is not about any one
model. It is that this repo could print a wrong cost figure and ship it, which
means the next wrong cost figure will also ship, and the next one may land on a
decision where $63 is not the stake.

Notice also the shape of the errors. Neither made a tier look absurd. The Haiku
miss made the cheapest model look *reasonable*, and the Sonnet cold writes made
the middle tier look *pointless*. Errors that produce implausible output get
caught. These produced plausible stories, which is why they survived.

**Q1b. Where would this have been caught first?**

A **startup check**, and the gap is not effort — it is timing and knowledge.

What it would have to know that a test does not: *its own configuration at the
moment it boots*. The service knows its model and can measure its prefix in one
free `countTokens` call. That pairing is the entire bug, and it is knowable
before the first request is served. A test only ever asserts what someone
already thought to check, and the whole failure here is that nobody thought to
check the cheap tier — smoke would have caught this on day one if anyone had
run it with `TRIAGE_MODEL` set, and for months nobody did.

Ranked by how early each instrument fires:

| instrument | when it fires | why it missed |
|---|---|---|
| startup check | boot, every deploy | **now exists** — `src/lib/preflight.ts`, added because of this bug |
| CI matrix | on the PR that changes the tier | smoke runs against the default model only |
| code review | on the diff | nobody memorizes per-model cache minimums |
| dashboard | after you are already paying | `cache_hit_rate` flat at zero is unmissable *if* someone looks |
| smoke test | only when someone runs it with the right env var | nobody did |

The honest ranking puts the test **fifth**. That is worth sitting with, because
the test is the thing this lab just spent a page improving. Making it fail
loudly was right — it now tells the truth to whoever runs it — but a correct
assertion nobody triggers is not a control. It is documentation with a CI
badge.

The transferable rule: **a check that requires someone to have already
suspected the problem is not a control.** Controls fire on their own schedule,
not on your suspicion. Startup and CI qualify; a manual smoke run does not.

**What was actually built.** [`src/lib/preflight.ts`](https://github.com/mrlynn/claude-triage-api/blob/main/src/lib/preflight.ts)
runs on every boot, measures the frozen prefix against the configured model's
minimum with one free `countTokens` call, and prints a loud block when caching
is off. Three details are worth copying into your own version:

1. **It measures `system[0]` only** — the frozen block that holds the
   breakpoint. Counting the whole request would include the volatile block and
   the user message, which sit after the breakpoint and do not count toward the
   minimum. That version passes while the real prefix falls short: a check that
   measures the wrong span is worse than no check, because it also removes the
   suspicion that something needs checking.
2. **It never blocks or throws.** It fires after the listener is up and is not
   awaited. A diagnostic that can take production down is a worse bug than the
   one it diagnoses. On any failure — offline, bad key, API error — it says it
   could not check and returns, and `willCache: false` is always paired with an
   `error` so "unknown" is never read as "confirmed broken".
3. **It names the wrong remedy explicitly.** The obvious reading of "prefix too
   short" is "make the prefix longer", which here means padding a legal
   document with ~1,300 tokens to win a discount. The message says not to.

Note what it still does not do: it warns, it does not exit non-zero. That is
deliberate for a dev server and arguably wrong for a deploy pipeline. Deciding
where that line sits for your own service is the actual exercise.

**Q1. What does the headroom do to the tier argument?**

It removes it. At 4,100 tickets a week the flagship costs about **$111 a
month** warm (the twelve-case matrix projects $132, cold writes included)
against a **$4,000** budget. Sonnet saves roughly $63 of that — under 2% of the
budget, in exchange for losing one to three more cases in twelve. The argument
does not improve when you give it its best case, which is the sign that it was
never a cost argument.

The general form is worth keeping: **cost optimization only matters where cost
is a binding constraint.** Northwind's binding constraint is the mis-routing
rate and the safety SLA, not the model bill. A team that optimizes the
non-binding constraint has done work that cannot show up in any outcome they
care about, and has spent accuracy to do it.

Where the argument *would* bite: raise volume to 400,000 tickets a week and
Opus becomes ~$10,800/month, over budget, while Sonnet is ~$4,600, and the
tradeoff is live again. Latency will not rescue it: in the 2026-09-15 run Sonnet
and Opus are within a second of each other at p50 (2.7s versus 3.3s).

So the honest recommendation for *this* system is: ship the flagship, and put
the effort you would have spent on tiering into the eval set instead.

**Q2. Is it a 5% problem?**

Much larger, and the accuracy number actively hides it.

The gold set is deliberately adversarial: twelve cases chosen because they sit
where rules touch. Real traffic is not distributed that way — most tickets are
"where is my package." So the cheap tier's *aggregate* accuracy on live traffic
would be far better than 8/12 or 9/12 suggests, and a naive rollout would look fine.

But the cases it loses are not randomly drawn from the distribution. They are
concentrated in exactly the population you built the system for. `eval-04` is a
child swallowing part of a product. Losing that case is not 8% of an accuracy
score; it is the failure mode from the October 2025 incident on
[the scenario page](https://triage.mlynn.dev/docs/scenario), reproduced by choice.

To answer properly you would need the joint distribution: how often does live
traffic hit a two-rule case, and what does being wrong on one cost? Northwind
can estimate the first from a sample of the archive and the second from the
handbook's SLA penalties. That is the measurement, and it is a business
measurement rather than a model one.

The trap to name out loud: **aggregate accuracy on a representative sample and
aggregate accuracy on an adversarial sample answer different questions, and
neither answers "what does this cost me."**

**Q3. What does a threshold catch when the gap is zero?**

Roughly its base rate, which is to say nothing useful.

If confidence on wrong answers is drawn from the same distribution as
confidence on right ones, then thresholding at 0.7 selects ~the same fraction
of each. You escalate some wrong answers, you escalate just as many right ones,
and you pay for a second call on all of them. The precision of the mechanism is
the model's error rate — no better than escalating at random.

In the two earlier runs where Haiku's gap came out negative, it is worse than random:
the threshold preferentially escalated cases the model had gotten *right*,
while passing the wrong ones straight through. You are paying a premium to
double-check the answers that did not need it.

The cost is not theoretical. Escalation on a low-signal model converges toward
"run every ticket twice," at which point you are paying the cheap tier plus the
flagship for a result no better than the flagship alone — strictly worse than
just calling the flagship.

**The rule that transfers: measure the calibration gap before you build
anything that routes on confidence.** It is one column in an eval you are
already running, and it is the difference between a control and a costly no-op.

**Q4. What message defeats `pickModel`?**

Any high-stakes ticket written without high-stakes vocabulary. The routing
signal is a keyword list, and the list can only match words the customer
happens to use.

The canonical example is already in this course: *"probably nothing, but the
bottle cap cracked and my kid swallowed a bit of plastic."* That one routes
correctly, because "swallowed" and "kid" are both on the list. Now write it the
way a worried, apologetic parent actually writes at 11pm: *"Hi — the lid on the
32oz came apart and some of it ended up in my daughter's mouth. She seems okay.
Just thought you should know."* No "swallow," no "injury," no "child." So it routes to
Sonnet, the cheap tier. Sonnet passed `eval-04` in the 2026-09-15 run, but the
router has just sent the case where being wrong is most expensive to the model
whose confidence you trust least.

It does not need an attacker. The failure mode is *politeness*.

**Why the failure is quiet:** nothing errors. The router logs a confident
`reason` ("no high-stakes language"), the cheap model returns a
well-formed schema-valid classification with 0.9 confidence, and the ticket goes
in the normal queue. Every component reports success. The only artifact is a
`meta.routed.reason` in a log nobody reads, and you find out when the safety
SLA is missed — which is precisely the three-day delay from the scenario.

Two consequences worth stating:

1. **A keyword router is a defence-in-depth layer, never the only layer.**
   `requires_human` is decided by the model reading the whole message, not by
   `pickModel`, and that ordering is deliberate.
2. **Bias the router toward escalation.** The `HIGH_STAKES` list is
   over-broad on purpose. A false positive costs a fraction of a cent; a false
   negative costs the incident.

Lab 8 makes the adversarial version of this explicit, where the message is
written to route itself down deliberately.

**Q5. What is escalation worth?**

On the cheap tier: close to nothing, for the reasons in Q3. Its confidence does
not predict its errors, so the trigger fires on the wrong population.

On Sonnet: possibly something, and you do not know yet. Across four earlier
runs its gap was 0.20–0.30 — not flagship, but informative enough that a 0.7
threshold selects disproportionately for wrong answers, which would make
`sonnet → escalate to opus` a defensible architecture. The 2026-09-15 run put it
at 0.05, with a wrong answer at 0.90 confidence. Twelve cases cannot tell you
which is the real number. Before shipping escalation from Sonnet, run the matrix
five times and look at the distribution of the gap, not a single value.

The general shape: **escalation is only as good as the calibration of the model
you escalate *from*.** It is not a safety net you can bolt onto any tier; it is
a mechanism that consumes a signal, and you have to check the signal exists.

Which reframes Steps 4 and 5 as complementary rather than alternative.
`pickModel` routes on the *input* and works regardless of the model's
self-knowledge, so it is what protects the safety case. Escalation routes on
the *output* and needs a calibrated model, so it is what catches genuine
ambiguity. Neither substitutes for the other, and a system that ships only the
second one has a hole exactly where it can least afford one.

**Q6. Two models score 11/12. Is that the same number?**

No, and there are three separate reasons, in increasing order of how much they
should worry you.

*Noise.* This set moves by up to two cases run-to-run with nothing changed. Two
observations of 11/12 are consistent with true rates anywhere from roughly 75%
to 100%. A twelve-case set cannot resolve a difference smaller than about eight
points, so "11/12 versus 11/12" is not evidence of equality — it is an absence
of evidence about anything.

*Composition.* Even if both are truly 11/12, they may be missing different
cases. One misses the deliberately ambiguous ticket; one misses the safety
report. Same score, and you would ship them into different jobs.

*What the score omits.* Accuracy is one column. Two models tied on it can differ
by a factor of ten in calibration gap — and Step 3 established that the gap
decides whether you can build a control on top of the model at all. The tie is
in the least informative column.

To distinguish them properly: run each five times and compare distributions
rather than points ($0.45 and the cheapest real improvement available), read the
disagreement matrix, and compare calibration gaps. If they still look identical,
you have learned something useful — pick on latency or price and stop
deliberating.

## Extension notes

Predict before you measure. Classification and prose generation load different
capabilities, and there is no rule that says a model that classifies worse also
writes worse. The cheap tiers may well write acceptable customer replies while
being materially worse at deciding what the reply should *say* — in which case
the correct architecture is not one tier for everything but a cheap drafter
behind an expensive classifier, which is a conclusion you can only reach by
measuring the two tasks separately.

Keep the judge pinned when you do it. The temptation to let each tier grade its
own prose is strong and it destroys the comparison.

Note also what you will be measuring: the judge itself swings 1/4 to 3/4 on
identical input. With a four-case sample you cannot distinguish tiers at all.
Raising the sample is most of the work, and deciding how far to raise it is
[Lab 6](https://triage.mlynn.dev/docs/labs/lab-6-evals) Q7 all over again.

**Q7. What is the latency column worth here?**

For Northwind's triage queue: **almost nothing, and saying so is the point.**
4,100 tickets a week arrive into a queue nobody reads in real time. Whether a
classification takes 3 seconds or 30 makes no difference to any human — the
tickets are processed faster than they arrive at every tier, and the constraint
that binds is accuracy on the cases where two handbook rules interact. Shipping
Sonnet to save 0.6 seconds nobody experiences, at the price of one to three
cases in twelve, is the same mistake as shipping it to save $63 against a
$4,000 budget. Same error, different column.

That is the transferable habit and it is worth stating flatly: **optimize the
constraint that binds.** A number being real, measured, and printed in your
table does not make it a decision input. Most of the columns in most model
comparisons are like this, and the discipline is knowing which one is yours
before you look.

*Now change the product.* Put the classifier in front of a person and latency
stops being decoration:

- **The storefront support form.** A customer watches the pipeline run while
  they wait. Here the p95 column matters (5.5s on Opus, 4.4s on Sonnet), and
  the classification is not what they came for — they came to file a ticket.
  The gap between tiers is small enough that the answer may be neither: stream
  a provisional result, or classify after submission. Measure p95 on the real
  form before deciding.
- **An agent-assist panel** that classifies while a human reads the ticket. The
  budget is however long the person spends reading, which is a few seconds. p95
  is the number that matters, not p50, because the failure is "the panel was
  still empty when I finished reading" and that happens in the tail.
- **A phone IVR routing a live caller.** Nothing above is shippable; you need a
  different architecture, not a different tier.

Two things to carry out of that. Which *percentile* you read is part of the
decision — p50 tells you what the experience usually is, p95 tells you how
often it is bad, and an interactive surface is judged on the tail. And a
system with both workloads should not pick one row: Northwind's real answer is
the flagship on the batch queue and the cheap tier on the live form, which is
`pickModel` from Step 4 doing exactly the job it exists for.

The measurement caveats matter before you quote any of this. These are
four-in-flight, whole-request times through the local route, not
time-to-first-token and not single-request figures. `/v1/triage` does not
stream; a streaming surface would be judged on time-to-first-token instead, and
`/v1/draft` would need its own measurement that this matrix does not make.

Source: https://triage.mlynn.dev/docs/solutions/lab-7
