Prompt caching and evals

Prompt caching lowers input cost when many requests share a long, stable prefix such as a policy handbook. Evals tell you whether the change preserved the behavior you need. Use them together: caching without measurement can hide a cost regression, and evaluation without usage data can miss the cost of an otherwise correct system.

How prompt caching works

Cache the stable prefix, then place changing context after the breakpoint:

system: [
  {
    type: "text",
    text: `${ROLE_INSTRUCTIONS}\n\n${POLICY_HANDBOOK}`,
    cache_control: { type: "ephemeral" },
  },
  { type: "text", text: `Channel: ${ticket.channel}` },
]

The cache is a byte-for-byte prefix match. A timestamp, reordered tools, or volatile customer context ahead of the breakpoint can silently turn every cache hit into a miss. HTTP 200 and a correct answer do not mean the cache works; inspect cache_read_input_tokens.

A minimal evaluation loop

1. Save a baseline across representative cases before changing the prompt or model.
2. Include safety-critical, ambiguous, adversarial, and multi-intent cases—not just easy examples.
3. Score deterministic requirements against explicit expected fields.
4. Use an LLM judge for subjective quality only with a concrete rubric and evidence requirements.
5. Compare quality, latency, and usage before promoting the change.

Two costly misconceptions

• “Caching always saves money.” A cache write costs more than uncached input; it pays off only when the prefix is reused within its lifetime.
• “An LLM judge is a CI gate.” Judges vary. Validate the rubric and treat a small, subjective judge as diagnostic evidence unless you have established its reliability.

Go deeper

Lab 5: prompt caching and cost has you break a cache four ways and read the accounting. Lab 6: eval design and LLM-as-judge turns the course’s critical failures into a test suite. The cache inspector is the fastest no-setup introduction.