# Lab 0 answers

**Q1. Why keep the gold set at twelve cases?**

Because every case is hand-labelled, and hand-labelling degrades with volume.
At twelve you can hold the whole set in your head, argue about each label, and
notice when two of them contradict each other. At two hundred you cannot, so in
practice the labels get produced faster and checked less — often by generating
them from a model, which quietly turns your eval into a measurement of
agreement with a previous model rather than of correctness.

The other loss is that a large set stops being read. A twelve-case failure
report is something you actually look at line by line; a two-hundred-case one
becomes a percentage.

None of which means twelve is *enough*. It is enough to catch a regression that
breaks a whole category, and not enough to support a claim about a rate. That
tradeoff is the honest position, and Lab 6 Q7 makes you size a set that can
support the rate claim instead.

**Q2. Which ticket did you disagree on?**

Most rooms disagree on `NW-T-1060` — the multi-intent one, where the customer
asks for a refund *and* an account email change in the same message, with a
mistyped order id. There is no wrong answer available: the message genuinely
contains two intents and the schema has one `category` field.

If you disagreed about `NW-T-1060`, the disagreement was about the **schema**,
not the ticket. That is the useful realization. A single-label classifier
applied to multi-intent input has a modelling problem, not an accuracy problem,
and no amount of prompt tuning fixes it — you either add a second field, route
on the primary intent and accept the loss, or split the ticket upstream.

`NW-T-1045` (flaking bottle lining, child's stomach upset) is the one people
get right but for varying reasons. Some label it `safety` because a child was
affected; some because handbook §5.4 says a safety report is logged even when
the customer downplays it. Both land on the same label here, but they diverge
on the next ticket, which is why the *reason* is worth surfacing out loud.

**Q3. Why is the confidence gap better than either number alone?**

Because a single mean confidence tells you about the model's manner, not its
accuracy. A model that reports 0.95 on everything has a high mean and a gap of
roughly zero, and its confidence field is decoration: you cannot threshold on
it, because thresholding requires that low scores predict something.

The gap says: when this model is wrong, does it *know*? A measured gap of
**~0.29** (0.87 on passes, ~0.58 on failures) means a confidence threshold
routes real work — set it at 0.6 and you catch most errors while sending most
tickets straight through. A gap near zero means a threshold either catches
nothing or blocks everything.

This is also why the calibration instruction in `TriageSchema`'s `.describe()`
text is load-bearing rather than documentation. Delete it and the gap collapses
while accuracy barely moves, which is the Lab 2 Step 2 experiment.

**Q4. What does the flip pattern tell you about confidence as a control?**

That it is a real signal and an insufficient one, and the two flipping cases
show both halves.

`eval-11` fails at ~0.46. A threshold at 0.6 routes it to a human, which is
exactly right — the ticket is genuinely multi-intent and no single confident
label exists. Confidence earned its keep.

`eval-03` fails at ~0.71. The same threshold passes it straight through. The
model is not hedging; it read "nine days" and applied an elapsed-day rule where
clause 3.2 specifies business days, and it is about as sure of that as it is of
the cases it gets right. A wrong answer delivered confidently is the failure
mode confidence cannot catch, because the score is the model's report on its
own state and that report is only as good as its reading of the rule.

So: threshold on confidence to catch ambiguity, and use something else to catch
misapplied rules. "Something else" here means a deterministic check — the
projected delivery date and today's date are both known, business days are
countable in code, and code does not have opinions about clause 3.2. That is
the argument Lab 8 generalizes when it moves refund authority out of a
model-judged boolean and into `enforceAuthority`.

**Q5. What is the smallest detectable improvement?**

Roughly **two cases, or 17 percentage points** — because that is the observed
run-to-run spread with nothing changed. Anything smaller is inside the noise,
and a single run showing 11/12 after a change that previously showed 10/12 is
not evidence of anything.

Three ways out, in increasing order of cost:

1. **Run it more than once.** Five runs of the current set costs about $0.45
   and turns one number into a distribution. Cheapest real improvement
   available, and almost nobody does it.
2. **Grow the set.** Variance falls roughly as the square root of n, so
   detecting a 5-point difference wants somewhere near a hundred cases — and
   now you are back to the labelling-quality problem in Q1.
3. **Stop measuring a rate.** Often you do not need "accuracy improved 4%."
   You need "the business-day rule now holds," which is a targeted assertion on
   one case class and needs no statistics at all.

Option 3 is usually the right answer and is usually reached last.

**Q6. Should `summary` be scored?**

*For not scoring it:* there is no single correct summary. Any automated check
would compare strings — exact match, embedding similarity, ROUGE — and all
three measure resemblance to one particular phrasing rather than correctness. A
summary that is accurate and differently worded would fail, which trains you to
edit the prompt toward the reference wording rather than toward good summaries.
That is overfitting to the metric, with extra steps.

*Against:* "unscored" means "unprotected." Nothing in this repo would notice if
`summary` started returning the empty string, or echoing the raw ticket, or
leaking the customer's card digits. It ships to a human reader in the ops
queue, so it has real failure modes and zero coverage.

*What to ship:* not a similarity score. Ship cheap invariants — non-empty, under
a length cap, contains no digit sequence longer than four, is not a substring of
the input — and leave quality to the LLM judge in Lab 6, which is the right
instrument for graded prose and is correctly kept out of the CI gate. The
lesson that transfers: the choice is not "score it" versus "don't." It is
"assert the properties you can state precisely, and judge the rest separately."

**Q7. Which eval belongs in CI?**

`eval:quick --gate`. Note that the reason is **not** cost — the scoreboard runs
about $0.09 against roughly $0.20 for the full harness, which is not a
difference worth designing around. The reason is determinism. `eval:quick`
compares enum values with `!==`, so the same input produces the same verdict,
and a red build points at a specific named case.

`npm run eval` includes the LLM judge, and the judge is too noisy to gate on —
three runs of this repo scored the tone judge at 3/4, then 1/4, then 2/4 on the
same four cases with nothing changed between runs. A 50-point spread cannot
detect a real change of any plausible size, so gating on it produces red builds
that mean nothing and teaches everyone to re-run until green.

The judge still earns its place; it just earns it in a different job. Run it
when you want to understand a result, not when you want a pass/fail. Lab 6
Q1 is where this gets argued properly.

## Extension notes

The instinct when writing a thirteenth case is to write one the system passes.
Push against it. A case that always passes contributes nothing but runtime — it
cannot go red, so it can never tell you anything.

Good candidates for a failing case: a ticket whose category is genuinely
contested (multi-intent, like `NW-T-1060`), one where the correct urgency
depends on a business-day calculation rather than an elapsed-day one
(`eval-03`'s trick), or a safety report phrased so casually that the safety
language never appears.

Whatever you write, fill in `notes` with the rule the case tests. Six months
from now that field is the only thing that distinguishes a real regression from
someone disagreeing with your label — and `eval:quick` prints it under every
failure precisely so that distinction is in front of you at the moment it
matters.

Source: https://triage.mlynn.dev/docs/solutions/lab-0
