Skip to main content

Labs

Eleven labs in sequence. Each one introduces exactly one new idea and depends on the one before it, so the order is the pedagogy rather than a suggestion.

Labs
11
Total time
7h 20m
Routes built
4
Start with Lab 0

Groundwork

Before any prompt work: a number to measure against, and the four fields that price everything else.

  1. Lab 0Establish your evaluation baselineHand-label three real tickets and record the score. Every change after this gets measured against that number.20 minbaselineSetup complete
  2. Lab 1Your first call, and reading usageOne call, four numbers. The usage block is the entire budget conversation, and it is invisible in the response body.20 minusageNode 20+, an Anthropic API key

The four capabilities

One route per idea. Each lab introduces exactly one new thing and builds on the one before it.

  1. Lab 2Structured outputs and schema designA schema is where you decide what the system may say, what it must commit to, and how it is required to express doubt.35 minPOST /v1/triageBuilds on Lab 1
  2. Lab 3Tool use and the agentic loopThe model looks up the order instead of believing the customer's account of it, and returns the trace that proves it did.45 minPOST /v1/resolveBuilds on Lab 2
  3. Lab 4Streaming and SSETokens arrive as they are generated, proxies buffer them, and the error lands after the status code already said 200.30 minPOST /v1/draftBuilds on Lab 1
  4. Lab 5Prompt caching and costThe handbook goes out on every request. Caching takes that line from about $765 a month to about $75, and every way of breaking it fails silently.35 mincache_controlBuilds on Labs 1 and 2

Judgment

What belongs on the scoreboard, and which model is worth spending the budget on.

  1. Lab 6Eval design and LLM-as-judgeWrite the cases you suspect will fail, starting with the one that opens “probably nothing,” then score what no assertion can.45 minLLM-as-judgeBuilds on Labs 0, 2 and 3
  2. Lab 7Choosing a modelMeasure before you downgrade. The five points a cheaper model gives up are disproportionately the cases that matter.45 minPOST /v1/estimateBuilds on Labs 0, 2 and 6

Production

The parts that separate a demo from a service someone depends on.

  1. Lab 8The trust boundaryA boolean the model wrote is not a permission check. Move the authority into code, then try to talk your way past it.60 minprompt injectionBuilds on Labs 0, 3 and 6
  2. Lab 9Shipping itBatches are half price and the measurement disagrees anyway. Rate limits, model pinning, and publishing tools to clients you did not write.60 minBatches APIBuilds on Labs 0, 3, 5 and 7
  3. Lab 10Ask NorthwindOne assistant across the course and the shop, with narrow typed tools and the write held behind a human confirmation.45 mincapstoneBuilds on Labs 3 and 8No API key