07 · Measuring Models
Evals
Tests you build for your own use
Beyond public benchmarks, the practical craft of writing your own tests for your own task — a graded set of real inputs and ideal outputs you re-run every time you change a prompt or model. Often a model grades the answers (“LLM-as-judge”). Increasingly its own job title.
Concrete example
Before shipping a support bot, you assemble 200 real tickets with ideal answers and score each model build against them.
Why it matters
The line between “it demoed well” and “it actually works” — and how teams catch regressions before users do.